2021-09-06T12:24:49.114 INFO:root:teuthology version: 1.1.0-df69d6db 2021-09-06T12:24:49.115 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857 --name soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi --verbose --description rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} --owner scheduled_soumyakoduri@teuthology -- /home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857/orig.config.yaml 2021-09-06T12:24:49.151 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-09-06T12:24:49.202 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857 branch: wip-skoduri-lua description: rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi} email: null first_in_suite: false job_id: '6376857' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi nuke-on-error: true os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-skoduri-lua ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync meta inject err probability: 0.1 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 rgw-multisite-tests: args: - tests.py - tests_ps.py selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-skoduri-lua sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 owner: scheduled_soumyakoduri@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 sleep_before_teardown: 0 suite: rgw suite_branch: wip-skoduri-lua suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 targets: smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKEAsKjpnirpPJZlrVlDB6EUcJevmYskuRXgQf6uMRHU+EU2BE/X82jOB1CqMpIfr8BU2Ji7R/5NNG5JscS8FgZ06sUAoAo8g06h6Qf4JVI/JcRlmNOALGPSbdOQVranDh1SQn+Jhs02F/M+I1xdxwd63mXbfaGGEe4YpIuYozFVISgrXDNjEm3BOxUP9BUhWNyzVIFjB4uLvNKqP03Zpp1amOEJ38pXu9IlCC1RMFhSqw7eWQDih957x+Pl8LbUBtQlq4b+XRrkSCL+OwdboW3keev60KLitsQqYpvGlAdynHhRrC0tWiMOmTQa4fEyzf6fZFZVAC9AYG1X7TIiQg/cVokcFCG2iymZBE0LXbzGRulXWiDE4orhmlt818fI+SCJoJ7exNpUpe/7H7bdLDncrY7KVO4fbaK8IGp32jmHMzYdMJ6yRw9Io25LtFFY/Obz0EZW4UpV+ccEpwtuFKR0qjvAQjUdurnqHU7dB5Ujdi4o4Xv/sgr2AO5KEChc= smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT1FWubehwPPALY5EhmmXs2T9Kf7zlSb7OkqXs4OzGfyd3xKEh38q10xEE4m6rgXG+GBpWHBJQvZUJSyBukInqY5N7EJ0/nzsyPQt/c7DcWSgvFuE4sA/ihpjg5ZikISU523gDbtc9AXqYX6wUouhh3LJ1ZI25LwHZQCkFON34Sii0I8wn58/9RtZY8AZ8C86UhbYZbVQvYwQJOYVPprf0rXSiQy4A1sZR4J/jm4oVU6VBtkzvzOBZNME011hmU5yWJ/RB6CPaf5nCO82HEBJorqfQ/QnJZeq3m5Jf87+mnOtitKe/ml8M6a0fcBq6XnGvPoD4o9bzRtrNg478pVxFOU+dbOI/YaV7zAtIWER2V5LRZgDzdmoFXBBxABf3h9PSiX3yPvT3PGbeNHAQctnGqLo1fYhBBKMt2mfxFkq/J5K+7GY8Z/Z1IlxE1qB479mOS+AiNxV6VwljlP5WIA2EkKjRAEvNjwr+xh12V2jzSNzH/IAYZ8R4/VfkmE6MUR8= tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master timestamp: 2021-09-06_10:30:05 tube: smithi user: soumyakoduri verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-09-06T12:24:49.203 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa; will attempt to use it 2021-09-06T12:24:49.204 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa/tasks 2021-09-06T12:24:49.204 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-09-06T12:24:49.206 INFO:teuthology.task.internal:Checking packages... 2021-09-06T12:24:49.225 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '0eb1a7943dd70e2a0b3086ea680284137a187e73' 2021-09-06T12:24:49.225 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-09-06T12:24:49.225 INFO:teuthology.packaging:ref: None 2021-09-06T12:24:49.226 INFO:teuthology.packaging:tag: None 2021-09-06T12:24:49.226 INFO:teuthology.packaging:branch: wip-skoduri-lua 2021-09-06T12:24:49.226 INFO:teuthology.packaging:sha1: 0eb1a7943dd70e2a0b3086ea680284137a187e73 2021-09-06T12:24:49.227 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-skoduri-lua 2021-09-06T12:24:49.393 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7472.g0eb1a794 2021-09-06T12:24:49.393 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-09-06T12:24:49.401 INFO:teuthology.task.internal:no buildpackages task found 2021-09-06T12:24:49.401 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-09-06T12:24:49.434 INFO:teuthology.task.internal:Saving configuration 2021-09-06T12:24:49.447 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-09-06T12:24:49.468 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-06T12:24:49.484 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-06 12:20:22.316857', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHJd4swEkF1zE4vtDhnbUtpJZIxM1rhkkL8A23cTfHdzlOR7Z4PZSsqASmMVWSr40pDnioZ69oiTt/r+0uETkmw=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi041.front.sepia.ceph.com'} 2021-09-06T12:24:49.498 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-09-06 12:20:22.316877', 'description': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857', 'locked_by': 'scheduled_soumyakoduri@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLLzHutBhm2Dl2LPA0/0MjRqxejkbGi0NLL0YOgbu88nkHQAx109M9ZR/gTWa3W5O6ea7c+7IUV/PjfF4kfygPc=', 'os_version': '8.4', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi052.front.sepia.ceph.com'} 2021-09-06T12:24:49.498 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-09-06T12:24:49.506 INFO:teuthology.task.internal:roles: ubuntu@smithi041.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-09-06T12:24:49.507 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-09-06T12:24:49.507 INFO:teuthology.run_tasks:Running task console_log... 2021-09-06T12:24:49.591 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f90bd842d90>, signals=[15]) 2021-09-06T12:24:49.592 INFO:teuthology.run_tasks:Running task internal.connect... 2021-09-06T12:24:49.599 INFO:teuthology.task.internal:Opening connections... 2021-09-06T12:24:49.599 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:24:49.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T12:24:49.672 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:24:49.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T12:24:49.746 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-09-06T12:24:49.753 DEBUG:teuthology.orchestra.run.smithi041:> uname -m 2021-09-06T12:24:49.770 INFO:teuthology.orchestra.run.smithi041.stdout:x86_64 2021-09-06T12:24:49.770 DEBUG:teuthology.orchestra.run.smithi041:> cat /etc/os-release 2021-09-06T12:24:49.826 INFO:teuthology.orchestra.run.smithi041.stdout:NAME="Red Hat Enterprise Linux" 2021-09-06T12:24:49.826 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION="8.4 (Ootpa)" 2021-09-06T12:24:49.826 INFO:teuthology.orchestra.run.smithi041.stdout:ID="rhel" 2021-09-06T12:24:49.826 INFO:teuthology.orchestra.run.smithi041.stdout:ID_LIKE="fedora" 2021-09-06T12:24:49.827 INFO:teuthology.orchestra.run.smithi041.stdout:VERSION_ID="8.4" 2021-09-06T12:24:49.827 INFO:teuthology.orchestra.run.smithi041.stdout:PLATFORM_ID="platform:el8" 2021-09-06T12:24:49.827 INFO:teuthology.orchestra.run.smithi041.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-06T12:24:49.827 INFO:teuthology.orchestra.run.smithi041.stdout:ANSI_COLOR="0;31" 2021-09-06T12:24:49.828 INFO:teuthology.orchestra.run.smithi041.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-06T12:24:49.829 INFO:teuthology.orchestra.run.smithi041.stdout:HOME_URL="https://www.redhat.com/" 2021-09-06T12:24:49.829 INFO:teuthology.orchestra.run.smithi041.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-06T12:24:49.829 INFO:teuthology.orchestra.run.smithi041.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-06T12:24:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:49.830 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-06T12:24:49.830 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-06T12:24:49.830 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-06T12:24:49.830 INFO:teuthology.orchestra.run.smithi041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-06T12:24:49.831 INFO:teuthology.lock.ops:Updating smithi041.front.sepia.ceph.com on lock server 2021-09-06T12:24:49.847 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2021-09-06T12:24:49.863 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2021-09-06T12:24:49.864 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2021-09-06T12:24:49.920 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Red Hat Enterprise Linux" 2021-09-06T12:24:49.921 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8.4 (Ootpa)" 2021-09-06T12:24:49.921 INFO:teuthology.orchestra.run.smithi052.stdout:ID="rhel" 2021-09-06T12:24:49.921 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="fedora" 2021-09-06T12:24:49.921 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8.4" 2021-09-06T12:24:49.922 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8" 2021-09-06T12:24:49.922 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-09-06T12:24:49.922 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2021-09-06T12:24:49.922 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-09-06T12:24:49.922 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.redhat.com/" 2021-09-06T12:24:49.923 INFO:teuthology.orchestra.run.smithi052.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-09-06T12:24:49.923 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-06T12:24:49.923 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:24:49.923 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-06T12:24:49.924 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-09-06T12:24:49.924 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-09-06T12:24:49.924 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-09-06T12:24:49.925 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2021-09-06T12:24:49.939 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-09-06T12:24:49.950 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-09-06T12:24:49.984 INFO:teuthology.task.internal:Checking for old test directory... 2021-09-06T12:24:49.985 DEBUG:teuthology.orchestra.run.smithi041:> test '!' -e /home/ubuntu/cephtest 2021-09-06T12:24:49.988 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2021-09-06T12:24:50.003 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-09-06T12:24:50.018 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-09-06T12:24:50.018 DEBUG:teuthology.orchestra.run.smithi041:> test -z $(ls -A /var/lib/ceph) 2021-09-06T12:24:50.043 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2021-09-06T12:24:50.071 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-09-06T12:24:50.110 INFO:teuthology.run_tasks:Running task kernel... 2021-09-06T12:24:50.125 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-09-06T12:24:50.126 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-09-06T12:24:50.126 DEBUG:teuthology.orchestra.run.smithi041:> uname -r 2021-09-06T12:24:50.143 INFO:teuthology.orchestra.run.smithi041.stdout:4.18.0-305.el8.x86_64 2021-09-06T12:24:50.143 INFO:teuthology.task.kernel:Running kernel on smithi041: 4.18.0-305.el8.x86_64 2021-09-06T12:24:50.143 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum install -y kernel 2021-09-06T12:24:51.553 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:24:51.553 INFO:teuthology.orchestra.run.smithi041.stdout:Unable to read consumer identity 2021-09-06T12:24:51.657 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:51.657 INFO:teuthology.orchestra.run.smithi041.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:24:51.657 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:54.379 INFO:teuthology.orchestra.run.smithi041.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 10 MB 00:01 2021-09-06T12:24:56.528 INFO:teuthology.orchestra.run.smithi041.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-09-06T12:24:57.109 INFO:teuthology.orchestra.run.smithi041.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-06T12:24:57.216 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:24:57.217 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:24:57.217 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:24:57.348 DEBUG:teuthology.orchestra.run.smithi041:> echo no | sudo yum reinstall kernel || true 2021-09-06T12:24:57.720 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:24:57.720 INFO:teuthology.orchestra.run.smithi041.stdout:Unable to read consumer identity 2021-09-06T12:24:57.773 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:57.773 INFO:teuthology.orchestra.run.smithi041.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:24:57.773 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:57.961 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:01 ago on Mon 06 Sep 2021 12:24:56 PM UTC. 2021-09-06T12:24:58.106 INFO:teuthology.orchestra.run.smithi041.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-06T12:24:58.143 INFO:teuthology.orchestra.run.smithi041.stderr:Error: No packages marked for reinstall. 2021-09-06T12:24:58.187 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum reinstall -y kernel || true 2021-09-06T12:24:58.518 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:24:58.518 INFO:teuthology.orchestra.run.smithi041.stdout:Unable to read consumer identity 2021-09-06T12:24:58.572 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:58.572 INFO:teuthology.orchestra.run.smithi041.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:24:58.572 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:24:58.758 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:02 ago on Mon 06 Sep 2021 12:24:56 PM UTC. 2021-09-06T12:24:58.881 INFO:teuthology.orchestra.run.smithi041.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-06T12:24:58.909 INFO:teuthology.orchestra.run.smithi041.stderr:Error: No packages marked for reinstall. 2021-09-06T12:24:58.957 DEBUG:teuthology.orchestra.run.smithi041:> rpm -q kernel --last | head -n 1 2021-09-06T12:24:59.030 INFO:teuthology.orchestra.run.smithi041.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-06T12:24:59.031 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-06T12:24:59.032 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-06T12:24:59.032 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2021-09-06T12:24:59.050 INFO:teuthology.orchestra.run.smithi052.stdout:4.18.0-305.el8.x86_64 2021-09-06T12:24:59.050 INFO:teuthology.task.kernel:Running kernel on smithi052: 4.18.0-305.el8.x86_64 2021-09-06T12:24:59.050 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2021-09-06T12:25:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:25:00.494 INFO:teuthology.orchestra.run.smithi052.stdout:Unable to read consumer identity 2021-09-06T12:25:00.589 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:00.590 INFO:teuthology.orchestra.run.smithi052.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:25:00.590 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:03.158 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 10 MB 00:01 2021-09-06T12:25:05.297 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 3.5 MB/s | 24 kB 00:00 2021-09-06T12:25:05.908 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-4.18.0-305.el8.x86_64 is already installed. 2021-09-06T12:25:06.021 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:25:06.022 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:25:06.023 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:25:06.174 DEBUG:teuthology.orchestra.run.smithi052:> echo no | sudo yum reinstall kernel || true 2021-09-06T12:25:06.549 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:25:06.549 INFO:teuthology.orchestra.run.smithi052.stdout:Unable to read consumer identity 2021-09-06T12:25:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:06.590 INFO:teuthology.orchestra.run.smithi052.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:25:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:06.778 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:01 ago on Mon 06 Sep 2021 12:25:05 PM UTC. 2021-09-06T12:25:06.902 INFO:teuthology.orchestra.run.smithi052.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-06T12:25:06.929 INFO:teuthology.orchestra.run.smithi052.stderr:Error: No packages marked for reinstall. 2021-09-06T12:25:06.973 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum reinstall -y kernel || true 2021-09-06T12:25:07.309 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:25:07.309 INFO:teuthology.orchestra.run.smithi052.stdout:Unable to read consumer identity 2021-09-06T12:25:07.354 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:07.354 INFO:teuthology.orchestra.run.smithi052.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2021-09-06T12:25:07.355 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:25:07.541 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:02 ago on Mon 06 Sep 2021 12:25:05 PM UTC. 2021-09-06T12:25:07.665 INFO:teuthology.orchestra.run.smithi052.stdout:Installed package kernel-4.18.0-305.el8.x86_64 (from anaconda) not available. 2021-09-06T12:25:07.687 INFO:teuthology.orchestra.run.smithi052.stderr:Error: No packages marked for reinstall. 2021-09-06T12:25:07.736 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel --last | head -n 1 2021-09-06T12:25:07.808 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-4.18.0-305.el8.x86_64 Thu 20 May 2021 08:07:00 PM UTC 2021-09-06T12:25:07.809 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.el8.x86_64 2021-09-06T12:25:07.810 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-09-06T12:25:07.810 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-09-06T12:25:07.810 DEBUG:teuthology.orchestra.run.smithi041:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-06T12:25:07.859 INFO:teuthology.orchestra.run.smithi041.stdout:ttyS1 2021-09-06T12:25:07.871 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-09-06T12:25:07.872 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-09-06T12:25:07.913 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2021-09-06T12:25:07.925 INFO:teuthology.run_tasks:Running task internal.base... 2021-09-06T12:25:07.934 INFO:teuthology.task.internal:Creating test directory... 2021-09-06T12:25:07.935 DEBUG:teuthology.orchestra.run.smithi041:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-06T12:25:07.937 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-09-06T12:25:07.981 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-09-06T12:25:07.991 INFO:teuthology.run_tasks:Running task internal.archive... 2021-09-06T12:25:08.032 INFO:teuthology.task.internal:Creating archive directory... 2021-09-06T12:25:08.033 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-06T12:25:08.036 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-09-06T12:25:08.068 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-09-06T12:25:08.078 INFO:teuthology.task.internal:Enabling coredump saving... 2021-09-06T12:25:08.079 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-06T12:25:08.105 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-09-06T12:25:08.131 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T12:25:08.137 INFO:teuthology.orchestra.run.smithi041.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T12:25:08.142 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T12:25:08.149 INFO:teuthology.orchestra.run.smithi041.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-09-06T12:25:08.151 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-09-06T12:25:08.161 INFO:teuthology.task.internal:Configuring sudo... 2021-09-06T12:25:08.162 DEBUG:teuthology.orchestra.run.smithi041:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-06T12:25:08.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-09-06T12:25:08.219 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-09-06T12:25:08.230 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-09-06T12:25:08.231 DEBUG:teuthology.orchestra.run.smithi041:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-06T12:25:08.261 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-09-06T12:25:08.276 DEBUG:teuthology.orchestra.run.smithi041:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T12:25:08.351 DEBUG:teuthology.orchestra.run.smithi041:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T12:25:08.420 DEBUG:teuthology.orchestra.run.smithi041:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T12:25:08.454 DEBUG:teuthology.orchestra.run.smithi041:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T12:25:08.504 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:25:08.505 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-06T12:25:08.571 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T12:25:08.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-09-06T12:25:08.695 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T12:25:08.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-09-06T12:25:08.779 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:25:08.779 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-09-06T12:25:08.844 DEBUG:teuthology.orchestra.run.smithi041:> sudo service rsyslog restart 2021-09-06T12:25:08.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2021-09-06T12:25:08.918 INFO:teuthology.orchestra.run.smithi041.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-06T12:25:08.941 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-09-06T12:25:09.297 INFO:teuthology.run_tasks:Running task internal.timer... 2021-09-06T12:25:09.307 INFO:teuthology.task.internal:Starting timer... 2021-09-06T12:25:09.307 INFO:teuthology.run_tasks:Running task pcp... 2021-09-06T12:25:09.341 INFO:teuthology.run_tasks:Running task selinux... 2021-09-06T12:25:09.378 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-09-06T12:25:09.378 DEBUG:teuthology.orchestra.run.smithi041:> sudo service auditd rotate 2021-09-06T12:25:09.447 INFO:teuthology.orchestra.run.smithi041.stdout:Rotating logs: [ OK ] 2021-09-06T12:25:09.449 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2021-09-06T12:25:09.526 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: [ OK ] 2021-09-06T12:25:09.528 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-09-06T12:25:09.529 DEBUG:teuthology.orchestra.run.smithi041:> /usr/sbin/getenforce 2021-09-06T12:25:09.549 INFO:teuthology.orchestra.run.smithi041.stdout:Permissive 2021-09-06T12:25:09.549 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2021-09-06T12:25:09.616 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2021-09-06T12:25:09.616 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi041.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi052.front.sepia.ceph.com': 'permissive'} 2021-09-06T12:25:09.617 DEBUG:teuthology.orchestra.run.smithi041:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-06T12:25:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T12:25:09.643 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-09-06T12:25:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T12:25:09.669 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-09-06T12:25:09.669 DEBUG:teuthology.orchestra.run.smithi041:> sudo /usr/sbin/setenforce permissive 2021-09-06T12:25:09.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2021-09-06T12:25:09.735 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-09-06T12:25:09.745 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-09-06T12:25:09.746 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-09-06T12:25:09.763 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-09-06T12:25:09.765 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi041.front.sepia.ceph.com,smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-09-06T12:25:10.116 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-09-06T12:25:10.874 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-09-06T12:25:13.623 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-09-06T12:25:13.644 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:13 +0000 (0:00:00.036) 0:00:00.036 ****** 2021-09-06T12:25:13.655 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:13 +0000 (0:00:00.010) 0:00:00.047 ****** 2021-09-06T12:25:15.084 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi041.front.sepia.ceph.com] 2021-09-06T12:25:15.131 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:01.476) 0:00:01.523 ****** 2021-09-06T12:25:15.161 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-09-06T12:25:15.206 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.074) 0:00:01.598 ****** 2021-09-06T12:25:15.398 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-09-06T12:25:15.399 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T12:25:15.420 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.214) 0:00:01.812 ****** 2021-09-06T12:25:15.438 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.017) 0:00:01.830 ****** 2021-09-06T12:25:15.816 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-09-06T12:25:15.817 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi041.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-09-06T12:25:15.839 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.400) 0:00:02.231 ****** 2021-09-06T12:25:15.857 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.018) 0:00:02.250 ****** 2021-09-06T12:25:15.905 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-09-06T12:25:15.907 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:15.927 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.069) 0:00:02.319 ****** 2021-09-06T12:25:15.934 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:15.956 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:15 +0000 (0:00:00.028) 0:00:02.348 ****** 2021-09-06T12:25:16.192 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-09-06T12:25:16.194 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:25:16.214 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:16 +0000 (0:00:00.258) 0:00:02.607 ****** 2021-09-06T12:25:16.231 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:16 +0000 (0:00:00.016) 0:00:02.624 ****** 2021-09-06T12:25:17.075 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-09-06T12:25:17.077 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630931116.2572637-21328-186186128021406/source", "state": "file", "uid": 0} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630931116.2755425-21330-278066077141294/source", "state": "file", "uid": 0} 2021-09-06T12:25:17.097 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:17 +0000 (0:00:00.866) 0:00:03.490 ****** 2021-09-06T12:25:17.116 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:17 +0000 (0:00:00.018) 0:00:03.509 ****** 2021-09-06T12:25:17.506 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-09-06T12:25:17.508 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-09-06T12:25:17.510 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-09-06T12:25:17.511 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-09-06T12:25:17.513 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-09-06T12:25:17.592 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-09-06T12:25:17.594 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-06T12:25:17.607 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Monday 06 September 2021 12:25:17 +0000 (0:00:00.490) 0:00:04.000 ****** 2021-09-06T12:25:18.146 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] 2021-09-06T12:25:18.164 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-09-06T12:25:18.179 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Monday 06 September 2021 12:25:18 +0000 (0:00:00.571) 0:00:04.572 ****** 2021-09-06T12:25:18.213 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-06T12:25:18.224 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-09-06T12:25:18.228 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-06T12:25:18.373 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:18 +0000 (0:00:00.193) 0:00:04.765 ****** 2021-09-06T12:25:18.394 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:18 +0000 (0:00:00.020) 0:00:04.786 ****** 2021-09-06T12:25:19.022 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi041.front.sepia.ceph.com] 2021-09-06T12:25:19.051 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-09-06T12:25:19.079 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-09-06T12:25:19.080 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:19 +0000 (0:00:00.685) 0:00:05.471 ****** 2021-09-06T12:25:19.123 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:19 +0000 (0:00:00.044) 0:00:05.516 ****** 2021-09-06T12:25:19.740 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-09-06T12:25:19.742 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.207442", "end": "2021-09-06 12:25:19.715190", "rc": 0, "start": "2021-09-06 12:25:19.507748", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.207350", "end": "2021-09-06 12:25:19.717288", "rc": 0, "start": "2021-09-06 12:25:19.509938", "stderr": "", "stderr_lines": [], "stdout": "540ef3733cffec190e79eb6099e95cef354b655b", "stdout_lines": ["540ef3733cffec190e79eb6099e95cef354b655b"]} 2021-09-06T12:25:19.762 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:19 +0000 (0:00:00.638) 0:00:06.154 ****** 2021-09-06T12:25:19.781 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:19 +0000 (0:00:00.019) 0:00:06.174 ****** 2021-09-06T12:25:19.963 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-09-06T12:25:19.965 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008049", "end": "2021-09-06 12:25:19.944510", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:25:19.936461", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-06T12:25:19.988 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:19 +0000 (0:00:00.207) 0:00:06.381 ****** 2021-09-06T12:25:20.007 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.034082", "end": "2021-09-06 12:25:19.990672", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:25:19.956590", "stderr": "", "stderr_lines": [], "stdout": "b2a91f5f527dcc382f8ed49697328a42af627ee8", "stdout_lines": ["b2a91f5f527dcc382f8ed49697328a42af627ee8"]} 2021-09-06T12:25:20.043 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.054) 0:00:06.435 ****** 2021-09-06T12:25:20.052 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-09-06T12:25:20.054 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.080 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.037) 0:00:06.473 ****** 2021-09-06T12:25:20.109 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.133 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.052) 0:00:06.525 ****** 2021-09-06T12:25:20.150 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-09-06T12:25:20.152 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.173 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.040) 0:00:06.566 ****** 2021-09-06T12:25:20.201 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.226 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.052) 0:00:06.618 ****** 2021-09-06T12:25:20.244 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-09-06T12:25:20.245 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.266 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.040) 0:00:06.659 ****** 2021-09-06T12:25:20.294 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.317 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.050) 0:00:06.709 ****** 2021-09-06T12:25:20.334 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-09-06T12:25:20.336 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.357 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.040) 0:00:06.750 ****** 2021-09-06T12:25:20.385 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.408 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.051) 0:00:06.801 ****** 2021-09-06T12:25:20.436 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-09-06T12:25:20.438 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.459 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.050) 0:00:06.851 ****** 2021-09-06T12:25:20.486 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:25:20.513 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.053) 0:00:06.905 ****** 2021-09-06T12:25:20.706 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-09-06T12:25:20.708 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:25:20.736 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.223) 0:00:07.129 ****** 2021-09-06T12:25:20.764 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:25:20.791 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:25:20 +0000 (0:00:00.054) 0:00:07.183 ****** 2021-09-06T12:25:21.037 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-09-06T12:25:21.039 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-06T12:25:21.081 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-09-06T12:25:21.237 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-06T12:25:21.281 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-09-06T12:25:21.437 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-06T12:25:21.490 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-09-06T12:25:21.647 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-06T12:25:21.690 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-09-06T12:25:21.847 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-06T12:25:21.900 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-09-06T12:25:22.046 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-06T12:25:22.110 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-09-06T12:25:22.257 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-06T12:25:22.310 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-09-06T12:25:22.456 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-06T12:25:22.511 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-09-06T12:25:22.656 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-06T12:25:22.719 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-09-06T12:25:22.865 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-06T12:25:22.928 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-09-06T12:25:23.065 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-06T12:25:23.128 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-09-06T12:25:23.263 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-06T12:25:23.337 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-09-06T12:25:23.462 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-06T12:25:23.536 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-09-06T12:25:23.672 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-06T12:25:23.746 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-09-06T12:25:23.871 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-06T12:25:23.955 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-09-06T12:25:24.080 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-06T12:25:24.153 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-09-06T12:25:24.279 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-06T12:25:24.352 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-09-06T12:25:24.488 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-06T12:25:24.562 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-09-06T12:25:24.687 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-06T12:25:24.771 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-09-06T12:25:24.896 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-06T12:25:24.971 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-09-06T12:25:25.096 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-06T12:25:25.169 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-09-06T12:25:25.295 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-06T12:25:25.378 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-09-06T12:25:25.503 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-06T12:25:25.576 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-09-06T12:25:25.702 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-06T12:25:25.787 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-09-06T12:25:25.912 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-06T12:25:25.986 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-09-06T12:25:26.112 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-06T12:25:26.189 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-09-06T12:25:26.357 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-06T12:25:26.390 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-09-06T12:25:26.557 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-06T12:25:26.599 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-09-06T12:25:26.755 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-06T12:25:26.797 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-09-06T12:25:26.964 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-06T12:25:26.996 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-09-06T12:25:27.163 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-06T12:25:27.196 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-09-06T12:25:27.363 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-06T12:25:27.405 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-09-06T12:25:27.572 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-06T12:25:27.604 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-09-06T12:25:27.771 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-06T12:25:27.813 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-09-06T12:25:27.971 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-06T12:25:28.014 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-09-06T12:25:28.171 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-06T12:25:28.224 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-09-06T12:25:28.380 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-06T12:25:28.423 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-09-06T12:25:28.589 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T12:25:28.632 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T12:25:28.788 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-06T12:25:28.831 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-09-06T12:25:28.998 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-06T12:25:29.032 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-09-06T12:25:29.200 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-06T12:25:29.242 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-09-06T12:25:29.399 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-06T12:25:29.451 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-09-06T12:25:29.607 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-06T12:25:29.650 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-09-06T12:25:29.806 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-06T12:25:29.848 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-09-06T12:25:30.014 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-06T12:25:30.056 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-09-06T12:25:30.212 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T12:25:30.264 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T12:25:30.421 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-06T12:25:30.463 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-09-06T12:25:30.621 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-06T12:25:30.663 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-09-06T12:25:30.819 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-06T12:25:30.871 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-09-06T12:25:31.027 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-06T12:25:31.070 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-09-06T12:25:31.226 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-06T12:25:31.279 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-09-06T12:25:31.426 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-06T12:25:31.478 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-09-06T12:25:31.624 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-06T12:25:31.678 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-09-06T12:25:31.834 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-06T12:25:31.887 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-09-06T12:25:32.033 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-06T12:25:32.086 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-09-06T12:25:32.232 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-06T12:25:32.295 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-09-06T12:25:32.441 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-06T12:25:32.494 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-09-06T12:25:32.645 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-06T12:25:32.698 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-09-06T12:25:32.844 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-06T12:25:32.896 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-09-06T12:25:33.043 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-06T12:25:33.096 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-09-06T12:25:33.252 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-06T12:25:33.305 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-09-06T12:25:33.450 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-06T12:25:33.503 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-09-06T12:25:33.660 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T12:25:33.713 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-09-06T12:25:33.860 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-06T12:25:33.912 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-09-06T12:25:34.059 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-06T12:25:34.112 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-09-06T12:25:34.267 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-06T12:25:34.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-09-06T12:25:34.466 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-06T12:25:34.519 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-09-06T12:25:34.675 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-06T12:25:34.717 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-09-06T12:25:34.872 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-06T12:25:34.925 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-09-06T12:25:35.071 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-06T12:25:35.124 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-09-06T12:25:35.281 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-06T12:25:35.333 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-09-06T12:25:35.479 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-06T12:25:35.532 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-09-06T12:25:35.677 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-06T12:25:35.740 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-09-06T12:25:35.886 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-06T12:25:35.939 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-09-06T12:25:36.085 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-06T12:25:36.148 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-09-06T12:25:36.284 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-06T12:25:36.348 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-09-06T12:25:36.484 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-06T12:25:36.547 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-09-06T12:25:36.685 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-06T12:25:36.759 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-09-06T12:25:36.894 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-06T12:25:36.957 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-09-06T12:25:37.093 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-06T12:25:37.156 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-09-06T12:25:37.291 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-06T12:25:37.364 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-09-06T12:25:37.500 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-06T12:25:37.564 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-09-06T12:25:37.699 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-06T12:25:37.773 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-09-06T12:25:37.898 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-06T12:25:37.971 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-09-06T12:25:38.107 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-06T12:25:38.171 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-09-06T12:25:38.306 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-06T12:25:38.379 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-09-06T12:25:38.516 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-06T12:25:38.580 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-09-06T12:25:38.715 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-06T12:25:38.789 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-09-06T12:25:38.914 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-06T12:25:38.987 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-09-06T12:25:39.123 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-06T12:25:39.186 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-09-06T12:25:39.321 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-06T12:25:39.394 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-09-06T12:25:39.530 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-06T12:25:39.593 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-09-06T12:25:39.728 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-06T12:25:39.802 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-09-06T12:25:39.927 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-06T12:25:40.001 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-09-06T12:25:40.136 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-06T12:25:40.199 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-09-06T12:25:40.334 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-06T12:25:40.408 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-09-06T12:25:40.533 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-06T12:25:40.606 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-09-06T12:25:40.742 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-06T12:25:40.816 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-09-06T12:25:40.940 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-06T12:25:41.014 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-09-06T12:25:41.139 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-06T12:25:41.212 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-09-06T12:25:41.347 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T12:25:41.421 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-09-06T12:25:41.546 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-06T12:25:41.619 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-09-06T12:25:41.744 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-06T12:25:41.829 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-09-06T12:25:41.954 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-06T12:25:42.027 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-09-06T12:25:42.152 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-06T12:25:42.236 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-09-06T12:25:42.350 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-06T12:25:42.433 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-09-06T12:25:42.558 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-06T12:25:42.641 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-09-06T12:25:42.756 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-06T12:25:42.840 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-09-06T12:25:42.966 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-06T12:25:43.040 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-09-06T12:25:43.165 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-06T12:25:43.239 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-09-06T12:25:43.375 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-06T12:25:43.438 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-09-06T12:25:43.575 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-06T12:25:43.649 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-09-06T12:25:43.774 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-06T12:25:43.847 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-09-06T12:25:43.973 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-06T12:25:44.047 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-09-06T12:25:44.173 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-06T12:25:44.257 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-09-06T12:25:44.383 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-06T12:25:44.456 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-09-06T12:25:44.582 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-06T12:25:44.666 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-09-06T12:25:44.782 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-06T12:25:44.866 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-09-06T12:25:44.993 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-06T12:25:45.067 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-09-06T12:25:45.192 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-06T12:25:45.275 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-09-06T12:25:45.401 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-06T12:25:45.474 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-09-06T12:25:45.600 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-06T12:25:45.673 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-09-06T12:25:45.809 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-06T12:25:45.883 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-09-06T12:25:46.019 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-06T12:25:46.082 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-09-06T12:25:46.219 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-06T12:25:46.282 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-09-06T12:25:46.418 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-06T12:25:46.492 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-09-06T12:25:46.628 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-06T12:25:46.702 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-09-06T12:25:46.829 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-06T12:25:46.913 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-09-06T12:25:47.028 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-06T12:25:47.114 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-09-06T12:25:47.238 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-06T12:25:47.322 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-09-06T12:25:47.448 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-06T12:25:47.533 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-09-06T12:25:47.648 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-06T12:25:47.732 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-09-06T12:25:47.847 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-06T12:25:47.934 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-09-06T12:25:48.060 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-06T12:25:48.144 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-09-06T12:25:48.261 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-06T12:25:48.346 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-09-06T12:25:48.462 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-06T12:25:48.546 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-09-06T12:25:48.661 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-06T12:25:48.746 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-09-06T12:25:48.874 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-06T12:25:48.959 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-09-06T12:25:49.074 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-06T12:25:49.159 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-09-06T12:25:50.760 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-06T12:25:50.885 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "name": "amctagga", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-09-06T12:25:52.735 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-06T12:25:52.871 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "name": "kevinz", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-09-06T12:25:54.376 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-06T12:25:54.501 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-09-06T12:25:55.991 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-06T12:25:56.147 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-09-06T12:25:57.622 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-06T12:25:57.871 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2021-09-06T12:25:59.328 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-06T12:25:59.453 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10107, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "name": "lflores", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10107} 2021-09-06T12:26:00.922 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-06T12:26:01.120 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10108, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10108} 2021-09-06T12:26:02.631 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-06T12:26:02.726 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10109, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10109} 2021-09-06T12:26:04.310 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-06T12:26:04.364 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10110, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "name": "luisrico", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10110} 2021-09-06T12:26:06.011 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-06T12:26:06.074 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10111, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "name": "matan", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10111} 2021-09-06T12:26:07.759 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-06T12:26:07.770 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10112, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10112} 2021-09-06T12:26:09.530 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-06T12:26:09.531 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10113, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10113} 2021-09-06T12:26:11.211 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-06T12:26:11.232 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10114, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10114} 2021-09-06T12:26:12.922 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-06T12:26:12.955 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10115, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10115} 2021-09-06T12:26:14.593 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-06T12:26:14.657 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10116, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10116} 2021-09-06T12:26:16.287 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-06T12:26:16.424 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10117, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "name": "melissali", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10117} 2021-09-06T12:26:18.031 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-06T12:26:18.148 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10118, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10118} 2021-09-06T12:26:19.770 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-06T12:26:19.875 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10119, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10119} 2021-09-06T12:26:21.523 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-06T12:26:21.637 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10120, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10120} 2021-09-06T12:26:21.878 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:21 +0000 (0:01:01.086) 0:01:08.270 ****** 2021-09-06T12:26:21.899 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:21 +0000 (0:00:00.021) 0:01:08.291 ****** 2021-09-06T12:26:21.941 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-09-06T12:26:21.963 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:21 +0000 (0:00:00.064) 0:01:08.355 ****** 2021-09-06T12:26:22.007 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:22 +0000 (0:00:00.043) 0:01:08.399 ****** 2021-09-06T12:26:22.047 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-09-06T12:26:22.049 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-06T12:26:22.051 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-06T12:26:22.053 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-09-06T12:26:22.054 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-09-06T12:26:22.056 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-09-06T12:26:22.058 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea 2021-09-06T12:26:22.058 INFO:teuthology.task.ansible.out:3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-09-06T12:26:22.060 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-09-06T12:26:22.061 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-09-06T12:26:22.063 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-09-06T12:26:22.065 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM 2021-09-06T12:26:22.065 INFO:teuthology.task.ansible.out:/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-09-06T12:26:22.067 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-09-06T12:26:22.069 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-09-06T12:26:22.070 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-09-06T12:26:22.072 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor- 2021-09-06T12:26:22.072 INFO:teuthology.task.ansible.out:mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": " 2021-09-06T12:26:22.074 INFO:teuthology.task.ansible.out:pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-06T12:26:22.102 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:22 +0000 (0:00:00.094) 0:01:08.494 ****** 2021-09-06T12:26:22.113 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-09-06T12:26:22.115 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-09-06T12:26:22.117 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-09-06T12:26:22.118 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-09-06T12:26:22.120 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1 2021-09-06T12:26:22.120 INFO:teuthology.task.ansible.out:eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-09-06T12:26:22.122 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-09-06T12:26:22.123 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-09-06T12:26:22.125 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-09-06T12:26:22.127 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-09-06T12:26:22.128 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-09-06T12:26:22.130 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-09-06T12:26:22.132 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0 2021-09-06T12:26:22.132 INFO:teuthology.task.ansible.out:fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-09-06T12:26:22.134 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-09-06T12:26:22.136 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": " 2021-09-06T12:26:22.137 INFO:teuthology.task.ansible.out:pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-09-06T12:26:22.163 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:22 +0000 (0:00:00.061) 0:01:08.555 ****** 2021-09-06T12:26:22.892 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-09-06T12:26:22.894 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} ok: [smithi041.front.sepia.ceph.com] => {"after": "540ef3733cffec190e79eb6099e95cef354b655b", "attempts": 1, "before": "540ef3733cffec190e79eb6099e95cef354b655b", "changed": false, "remote_url_changed": false} 2021-09-06T12:26:22.894 INFO:teuthology.task.ansible.out: 2021-09-06T12:26:22.916 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:22 +0000 (0:00:00.753) 0:01:09.309 ****** 2021-09-06T12:26:22.937 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:26:22 +0000 (0:00:00.020) 0:01:09.329 ****** 2021-09-06T12:26:23.268 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-09-06T12:26:23.269 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-06T12:26:23.311 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-09-06T12:26:23.520 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-06T12:26:23.563 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-09-06T12:26:23.763 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-06T12:26:23.795 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-09-06T12:26:24.005 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-06T12:26:24.006 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-06T12:26:24.047 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-09-06T12:26:24.049 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-09-06T12:26:24.239 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-06T12:26:24.292 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-09-06T12:26:24.479 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-06T12:26:24.481 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-06T12:26:24.532 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-09-06T12:26:24.534 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-09-06T12:26:24.718 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-06T12:26:24.781 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-09-06T12:26:24.970 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-06T12:26:25.034 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-09-06T12:26:25.238 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-06T12:26:25.251 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:25.282 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-09-06T12:26:25.293 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:25.512 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-06T12:26:25.514 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-06T12:26:25.544 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-09-06T12:26:25.546 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-09-06T12:26:25.754 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-06T12:26:25.756 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-06T12:26:25.765 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:25.776 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:25.798 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-09-06T12:26:25.799 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-09-06T12:26:25.808 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:25.809 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:26:26.030 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-06T12:26:26.063 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-09-06T12:26:26.284 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-06T12:26:26.286 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-06T12:26:26.306 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-09-06T12:26:26.308 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-09-06T12:26:26.517 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-06T12:26:26.570 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-09-06T12:26:26.759 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-06T12:26:26.761 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-06T12:26:26.813 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-09-06T12:26:26.815 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-09-06T12:26:27.002 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-06T12:26:27.054 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-09-06T12:26:27.253 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-06T12:26:27.295 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-09-06T12:26:27.493 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-06T12:26:27.535 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-09-06T12:26:27.733 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-06T12:26:27.735 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-06T12:26:27.785 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-09-06T12:26:27.787 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-09-06T12:26:27.985 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-06T12:26:27.987 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-06T12:26:28.049 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-09-06T12:26:28.051 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-09-06T12:26:28.228 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-06T12:26:28.281 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-09-06T12:26:28.468 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-06T12:26:28.521 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-09-06T12:26:28.708 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-06T12:26:28.710 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-06T12:26:28.771 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7 2021-09-06T12:26:28.773 INFO:teuthology.task.ansible.out:gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-09-06T12:26:28.948 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-06T12:26:29.011 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-09-06T12:26:29.199 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-06T12:26:29.252 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-09-06T12:26:29.439 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-06T12:26:29.513 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-09-06T12:26:29.679 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-06T12:26:29.763 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-09-06T12:26:29.932 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-06T12:26:30.006 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-09-06T12:26:30.174 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-06T12:26:30.176 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-06T12:26:30.247 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-09-06T12:26:30.249 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-09-06T12:26:30.414 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-06T12:26:30.499 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-09-06T12:26:30.655 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-06T12:26:30.739 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-09-06T12:26:30.895 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-06T12:26:30.999 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-09-06T12:26:31.134 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-06T12:26:31.239 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-09-06T12:26:31.375 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-06T12:26:31.479 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-09-06T12:26:31.625 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T12:26:31.719 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T12:26:31.864 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-06T12:26:31.968 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-09-06T12:26:32.105 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-06T12:26:32.212 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-09-06T12:26:32.348 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-06T12:26:32.350 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-06T12:26:32.452 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-09-06T12:26:32.454 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-09-06T12:26:32.588 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-06T12:26:32.590 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-06T12:26:32.692 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-09-06T12:26:32.694 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-09-06T12:26:32.837 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-06T12:26:32.932 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-09-06T12:26:33.077 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-06T12:26:33.171 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-09-06T12:26:33.317 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-06T12:26:33.412 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-09-06T12:26:33.559 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-06T12:26:33.561 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T12:26:33.654 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-09-06T12:26:33.656 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T12:26:33.810 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-06T12:26:33.812 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-06T12:26:33.813 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-06T12:26:33.894 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-09-06T12:26:33.896 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-09-06T12:26:33.897 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-09-06T12:26:34.061 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-06T12:26:34.145 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-09-06T12:26:34.302 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-06T12:26:34.387 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-09-06T12:26:34.546 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-06T12:26:34.630 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-09-06T12:26:34.786 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-06T12:26:34.870 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-09-06T12:26:35.036 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-06T12:26:35.112 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-09-06T12:26:35.279 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-06T12:26:35.281 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-06T12:26:35.353 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-09-06T12:26:35.354 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-09-06T12:26:35.520 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-06T12:26:35.593 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-09-06T12:26:35.763 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-06T12:26:35.836 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-09-06T12:26:36.003 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-06T12:26:36.005 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-06T12:26:36.077 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-09-06T12:26:36.079 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-09-06T12:26:36.257 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-06T12:26:36.331 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-09-06T12:26:36.499 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-06T12:26:36.573 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-09-06T12:26:36.741 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-06T12:26:36.742 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-06T12:26:36.815 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-09-06T12:26:36.817 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-09-06T12:26:36.983 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-06T12:26:37.056 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-09-06T12:26:37.219 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-06T12:26:37.303 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-09-06T12:26:37.481 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-06T12:26:37.544 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-09-06T12:26:37.722 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T12:26:37.785 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-09-06T12:26:37.962 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-06T12:26:38.026 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-09-06T12:26:38.204 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-06T12:26:38.267 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-09-06T12:26:38.445 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-06T12:26:38.508 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-09-06T12:26:38.684 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-06T12:26:38.747 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-09-06T12:26:38.925 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-06T12:26:38.988 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-09-06T12:26:39.165 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-06T12:26:39.228 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-09-06T12:26:39.406 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-06T12:26:39.470 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-09-06T12:26:39.649 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-06T12:26:39.713 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-09-06T12:26:39.889 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-06T12:26:39.952 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-09-06T12:26:40.131 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-06T12:26:40.196 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-09-06T12:26:40.376 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-06T12:26:40.439 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-09-06T12:26:40.616 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-06T12:26:40.690 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-09-06T12:26:40.855 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-06T12:26:40.928 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-09-06T12:26:41.095 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-06T12:26:41.168 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-09-06T12:26:41.335 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-06T12:26:41.419 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-09-06T12:26:41.585 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-06T12:26:41.659 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-09-06T12:26:41.836 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-06T12:26:41.838 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-06T12:26:41.910 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-09-06T12:26:41.912 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-09-06T12:26:42.076 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-06T12:26:42.149 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-09-06T12:26:42.316 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-06T12:26:42.389 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-09-06T12:26:42.578 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-06T12:26:42.631 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-09-06T12:26:42.819 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-06T12:26:42.872 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-09-06T12:26:42.874 INFO:teuthology.task.ansible.out: 2021-09-06T12:26:43.061 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-06T12:26:43.063 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-06T12:26:43.125 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-09-06T12:26:43.126 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-09-06T12:26:43.303 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-06T12:26:43.305 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-06T12:26:43.367 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-09-06T12:26:43.369 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-09-06T12:26:43.545 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-06T12:26:43.608 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-09-06T12:26:43.796 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-06T12:26:43.849 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-09-06T12:26:44.037 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-06T12:26:44.090 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-09-06T12:26:44.288 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-06T12:26:44.341 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-09-06T12:26:44.533 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-06T12:26:44.596 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-09-06T12:26:44.774 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-06T12:26:44.837 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-09-06T12:26:45.016 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-06T12:26:45.080 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-09-06T12:26:45.259 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-06T12:26:45.261 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-06T12:26:45.324 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-09-06T12:26:45.326 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-09-06T12:26:45.501 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-06T12:26:45.584 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-09-06T12:26:45.742 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-06T12:26:45.826 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-09-06T12:26:45.983 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-06T12:26:46.067 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-09-06T12:26:46.226 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-06T12:26:46.310 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-09-06T12:26:46.478 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-06T12:26:46.552 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-09-06T12:26:46.719 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-06T12:26:46.803 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-09-06T12:26:46.951 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-06T12:26:46.953 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T12:26:47.045 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-09-06T12:26:47.047 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-09-06T12:26:47.202 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-06T12:26:47.297 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-09-06T12:26:47.445 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-06T12:26:47.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-09-06T12:26:47.687 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-06T12:26:47.771 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-09-06T12:26:47.917 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-06T12:26:48.024 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-09-06T12:26:48.161 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-06T12:26:48.265 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-09-06T12:26:48.403 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-06T12:26:48.498 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-09-06T12:26:48.653 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-06T12:26:48.737 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-09-06T12:26:48.893 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-06T12:26:48.987 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-09-06T12:26:49.144 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-06T12:26:49.231 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-09-06T12:26:49.378 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-06T12:26:49.472 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-09-06T12:26:49.619 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-06T12:26:49.714 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-09-06T12:26:49.861 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-06T12:26:49.956 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-09-06T12:26:50.103 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-06T12:26:50.197 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-09-06T12:26:50.344 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-06T12:26:50.437 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-09-06T12:26:50.583 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-06T12:26:50.687 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-09-06T12:26:50.822 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-06T12:26:50.936 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-09-06T12:26:51.062 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-06T12:26:51.177 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-09-06T12:26:51.302 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-06T12:26:51.417 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-09-06T12:26:51.552 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-06T12:26:51.554 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-06T12:26:51.656 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-09-06T12:26:51.658 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-09-06T12:26:51.792 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-06T12:26:51.896 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-09-06T12:26:52.031 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-06T12:26:52.135 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-09-06T12:26:52.270 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-06T12:26:52.375 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-09-06T12:26:52.510 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-06T12:26:52.615 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-09-06T12:26:52.750 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-06T12:26:52.898 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-09-06T12:26:52.996 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-06T12:26:53.146 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-09-06T12:26:53.245 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-06T12:26:53.392 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-09-06T12:26:53.487 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-06T12:26:53.633 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-09-06T12:26:53.728 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-06T12:26:53.874 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-09-06T12:26:53.968 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-06T12:26:54.114 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-09-06T12:26:54.208 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-06T12:26:54.353 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-09-06T12:26:54.449 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-06T12:26:54.597 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-09-06T12:26:54.691 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-06T12:26:54.840 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-09-06T12:26:54.934 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-06T12:26:55.081 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-09-06T12:26:55.176 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-06T12:26:55.324 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-09-06T12:26:55.421 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-06T12:26:55.569 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-09-06T12:26:55.675 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-06T12:26:55.677 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-06T12:26:55.813 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-09-06T12:26:55.815 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-09-06T12:26:55.919 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-06T12:26:56.077 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-09-06T12:26:56.164 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-06T12:26:56.327 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-09-06T12:26:56.425 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-06T12:26:56.584 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-09-06T12:26:56.679 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-06T12:26:56.826 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-09-06T12:26:56.919 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-06T12:26:57.067 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-09-06T12:26:57.161 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-06T12:26:57.313 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-09-06T12:26:57.407 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-06T12:26:57.557 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-09-06T12:26:57.651 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-06T12:26:57.797 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-09-06T12:26:57.891 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-06T12:26:58.036 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-09-06T12:26:58.140 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-06T12:26:58.286 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-09-06T12:26:58.390 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-06T12:26:58.526 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-09-06T12:26:58.527 INFO:teuthology.task.ansible.out: 2021-09-06T12:26:58.630 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-06T12:26:58.633 INFO:teuthology.task.ansible.out: 2021-09-06T12:26:58.767 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8l7qvSQR7fnZdQ1yB1F1jgde0OmLcwcDFzc/o2dYBb matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-09-06T12:26:58.768 INFO:teuthology.task.ansible.out: 2021-09-06T12:26:58.872 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-06T12:26:59.017 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-09-06T12:26:59.121 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-06T12:26:59.256 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-09-06T12:26:59.357 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-06T12:26:59.504 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-09-06T12:26:59.598 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-06T12:26:59.744 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-09-06T12:26:59.841 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-06T12:26:59.988 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-09-06T12:27:00.083 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-06T12:27:00.230 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-09-06T12:27:00.334 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-06T12:27:00.470 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-09-06T12:27:00.575 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-06T12:27:00.720 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-09-06T12:27:00.825 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-06T12:27:00.916 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:00 +0000 (0:00:37.979) 0:01:47.308 ****** 2021-09-06T12:27:00.966 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-09-06T12:27:01.034 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-09-06T12:27:01.036 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.058 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:01 +0000 (0:00:00.141) 0:01:47.450 ****** 2021-09-06T12:27:01.066 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.066 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.068 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.082 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.084 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.135 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.146 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.157 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.158 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.168 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.169 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.179 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.190 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.192 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.319 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-06T12:27:01.330 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.341 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.446 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-09-06T12:27:01.457 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.468 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.572 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-06T12:27:01.697 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-09-06T12:27:01.812 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-06T12:27:01.823 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.834 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.835 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.845 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.847 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.857 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.858 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.868 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.879 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.880 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.890 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.892 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.901 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.912 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.914 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.923 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.925 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.935 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.937 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-09-06T12:27:01.946 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.948 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef 2021-09-06T12:27:01.950 INFO:teuthology.task.ansible.out:8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.959 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.960 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.970 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.972 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.983 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.984 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:01.996 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": fal 2021-09-06T12:27:01.997 INFO:teuthology.task.ansible.out:se, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.006 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.008 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.018 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.020 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.031 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.033 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.043 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.045 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.055 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.057 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.067 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.069 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.080 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.081 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.092 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.094 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.103 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.105 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.115 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.117 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.127 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.129 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.140 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.141 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.151 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.152 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.152 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.153 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.164 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.165 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.175 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.177 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.187 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.189 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.199 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.201 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.212 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.213 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.223 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.224 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.235 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.237 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.247 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.249 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.260 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.261 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.262 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.273 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.275 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.285 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.287 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.297 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.299 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.309 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.311 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.322 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.324 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.334 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.336 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.346 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.348 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.359 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.361 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.372 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.374 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.384 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.386 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.396 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.397 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.399 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.399 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.410 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.412 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.422 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.424 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.434 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.436 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.446 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.448 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.459 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.460 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.461 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.471 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.473 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.483 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.485 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.495 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.497 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.507 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.509 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.519 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.520 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.531 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.533 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.543 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.545 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.555 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.557 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.567 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.569 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.579 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.581 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.591 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.593 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.603 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.605 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.615 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.616 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.616 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.617 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.627 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.629 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.639 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.641 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.652 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.653 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.664 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.666 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.676 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.678 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.688 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.689 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.699 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.701 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.711 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.713 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.723 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.725 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.736 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.738 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.747 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.749 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.760 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.762 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.772 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.774 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.785 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.786 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.796 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.798 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.809 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.810 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.820 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.822 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.841 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.865 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:02 +0000 (0:00:01.806) 0:01:49.257 ****** 2021-09-06T12:27:02.872 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e\nmatan@thinkpad JJuwaWbac2lbcH5pCGON1w 521fe30e3a950915a1b538913c1af76198c4133018b7869d348740c708b38a7d\n"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.873 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.875 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.875 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.877 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.888 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.889 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.900 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.947 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:02 +0000 (0:00:00.082) 0:01:49.340 ****** 2021-09-06T12:27:02.954 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-09-06T12:27:02.955 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:02.979 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:02 +0000 (0:00:00.031) 0:01:49.371 ****** 2021-09-06T12:27:03.017 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:03.040 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:03 +0000 (0:00:00.061) 0:01:49.433 ****** 2021-09-06T12:27:03.264 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-09-06T12:27:03.266 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-06T12:27:03.338 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-09-06T12:27:03.463 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-06T12:27:03.536 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-09-06T12:27:03.661 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-06T12:27:03.734 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-09-06T12:27:03.859 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-06T12:27:03.932 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-09-06T12:27:04.057 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-06T12:27:04.130 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-09-06T12:27:04.264 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-06T12:27:04.328 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-09-06T12:27:04.463 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-06T12:27:04.528 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-09-06T12:27:04.663 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-06T12:27:04.726 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-09-06T12:27:04.862 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-06T12:27:04.925 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-09-06T12:27:05.063 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-06T12:27:05.137 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-09-06T12:27:05.262 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-06T12:27:05.336 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-09-06T12:27:05.461 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-06T12:27:05.535 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-09-06T12:27:05.660 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-06T12:27:05.734 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-09-06T12:27:05.849 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-06T12:27:05.933 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-09-06T12:27:06.048 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-06T12:27:06.133 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-09-06T12:27:06.248 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-06T12:27:06.332 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-09-06T12:27:06.446 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-06T12:27:06.530 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-09-06T12:27:06.646 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-06T12:27:06.729 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-09-06T12:27:06.844 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-06T12:27:06.928 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-09-06T12:27:07.044 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-06T12:27:07.128 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-09-06T12:27:07.244 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-06T12:27:07.328 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-09-06T12:27:07.434 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-06T12:27:07.528 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-09-06T12:27:07.634 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-06T12:27:07.729 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-09-06T12:27:07.834 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-06T12:27:07.927 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-09-06T12:27:08.033 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-06T12:27:08.118 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-09-06T12:27:08.233 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-06T12:27:08.317 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-09-06T12:27:08.433 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-06T12:27:08.516 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-09-06T12:27:08.631 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-06T12:27:08.714 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-09-06T12:27:08.829 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-06T12:27:08.912 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-09-06T12:27:09.026 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-06T12:27:09.110 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-09-06T12:27:09.223 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-06T12:27:09.318 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-09-06T12:27:09.422 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-06T12:27:09.505 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-09-06T12:27:09.619 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-06T12:27:09.713 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-09-06T12:27:09.818 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-06T12:27:09.911 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-09-06T12:27:10.016 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-06T12:27:10.110 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-09-06T12:27:10.214 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-06T12:27:10.319 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-09-06T12:27:10.412 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-06T12:27:10.517 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-09-06T12:27:10.621 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-06T12:27:10.715 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-09-06T12:27:10.819 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-06T12:27:10.913 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-09-06T12:27:11.017 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-06T12:27:11.121 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-09-06T12:27:11.205 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-06T12:27:11.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-09-06T12:27:11.414 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-06T12:27:11.518 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-09-06T12:27:11.612 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-06T12:27:11.716 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-09-06T12:27:11.810 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-06T12:27:11.924 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-09-06T12:27:12.007 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-06T12:27:12.122 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-09-06T12:27:12.205 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-06T12:27:12.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-09-06T12:27:12.403 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-06T12:27:12.518 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-09-06T12:27:12.602 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-06T12:27:12.716 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-09-06T12:27:12.810 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-06T12:27:12.852 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:12 +0000 (0:00:09.811) 0:01:59.244 ****** 2021-09-06T12:27:12.921 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-09-06T12:27:12.971 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:12 +0000 (0:00:00.119) 0:01:59.363 ****** 2021-09-06T12:27:13.402 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-06T12:27:13.404 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630931232.9423344-28431-109374143145997/source", "state": "file", "uid": 0} 2021-09-06T12:27:13.425 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.453) 0:01:59.817 ****** 2021-09-06T12:27:13.493 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi041.front.sepia.ceph.com] => { "msg": "Host smithi041.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-06T12:27:13.515 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.090) 0:01:59.907 ****** 2021-09-06T12:27:13.532 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-09-06T12:27:13.534 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "86e4a2dae5f62bb631669de9d1a6c6efe9b72935", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "5694a3125663295343f0bc40f8059adb", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1630931233.0583944-28439-7577699502602/source", "state": "file", "uid": 0} 2021-09-06T12:27:13.557 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.042) 0:01:59.950 ****** 2021-09-06T12:27:13.586 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-09-06T12:27:13.588 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-06T12:27:13.608 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.050) 0:02:00.001 ****** 2021-09-06T12:27:13.634 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-09-06T12:27:13.636 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => { "msg": "Host smithi052.front.sepia.ceph.com is running RedHat 8.4 (Ootpa)" } 2021-09-06T12:27:13.658 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.049) 0:02:00.050 ****** 2021-09-06T12:27:13.675 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-06T12:27:13.677 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:13.699 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.040) 0:02:00.091 ****** 2021-09-06T12:27:13.738 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-09-06T12:27:13.739 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2021-09-06T12:27:13.762 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.063) 0:02:00.155 ****** 2021-09-06T12:27:13.844 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-09-06T12:27:13.845 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:13.866 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:13 +0000 (0:00:00.103) 0:02:00.258 ****** 2021-09-06T12:27:13.998 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-06T12:27:14.000 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.065080", "end": "2021-09-06 12:27:13.978968", "rc": 0, "start": "2021-09-06 12:27:13.913888", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-09-06T12:27:14.021 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.154) 0:02:00.413 ****** 2021-09-06T12:27:14.090 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-06T12:27:14.092 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.112 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.090) 0:02:00.504 ****** 2021-09-06T12:27:14.192 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-09-06T12:27:14.194 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.075845", "end": "2021-09-06 12:27:14.167767", "rc": 0, "start": "2021-09-06 12:27:14.091922", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.215 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.102) 0:02:00.607 ****** 2021-09-06T12:27:14.234 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.018) 0:02:00.626 ****** 2021-09-06T12:27:14.282 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-06T12:27:14.284 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.304 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.070) 0:02:00.697 ****** 2021-09-06T12:27:14.312 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-09-06T12:27:14.314 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.336 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.031) 0:02:00.729 ****** 2021-09-06T12:27:14.374 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-06T12:27:14.376 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.398 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.061) 0:02:00.790 ****** 2021-09-06T12:27:14.417 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-09-06T12:27:14.419 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.440 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.042) 0:02:00.833 ****** 2021-09-06T12:27:14.479 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-06T12:27:14.481 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.503 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.061) 0:02:00.895 ****** 2021-09-06T12:27:14.520 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-09-06T12:27:14.522 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.544 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.041) 0:02:00.936 ****** 2021-09-06T12:27:14.581 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-09-06T12:27:14.583 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-06T12:27:14.605 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.061) 0:02:00.997 ****** 2021-09-06T12:27:14.624 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-09-06T12:27:14.626 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.648 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.042) 0:02:01.040 ****** 2021-09-06T12:27:14.697 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-06T12:27:14.698 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T12:27:14.720 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.072) 0:02:01.112 ****** 2021-09-06T12:27:14.737 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-09-06T12:27:14.739 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:14.761 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.040) 0:02:01.153 ****** 2021-09-06T12:27:14.799 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-06T12:27:14.801 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-06T12:27:14.826 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.065) 0:02:01.218 ****** 2021-09-06T12:27:14.843 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-09-06T12:27:14.845 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2021-09-06T12:27:14.867 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.041) 0:02:01.259 ****** 2021-09-06T12:27:14.956 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-09-06T12:27:14.958 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T12:27:14.979 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:14 +0000 (0:00:00.112) 0:02:01.371 ****** 2021-09-06T12:27:15.058 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-09-06T12:27:15.060 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2021-09-06T12:27:15.080 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:15 +0000 (0:00:00.100) 0:02:01.472 ****** 2021-09-06T12:27:16.138 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-09-06T12:27:16.140 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1630931235.9751062-ebb_hqp0/katello-ca-consumer-latest.noarch85tn8e8c.rpm"]} 2021-09-06T12:27:16.161 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:16 +0000 (0:00:01.081) 0:02:02.553 ****** 2021-09-06T12:27:16.189 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1630931236.037065-s94kdsf1/katello-ca-consumer-latest.noarch99e_qetx.rpm"]} 2021-09-06T12:27:16.213 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:16 +0000 (0:00:00.052) 0:02:02.606 ****** 2021-09-06T12:27:16.758 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* 2021-09-06T12:27:16.760 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "532cb5b70a3f71ce180c165205647e8909d4e8a0", "dest": "/etc/yum/vars/releasever", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/yum/vars/releasever", "secontext": "system_u:object_r:etc_t:s0", "size": 3, "state": "file", "uid": 0} 2021-09-06T12:27:16.781 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:16 +0000 (0:00:00.567) 0:02:03.174 ****** 2021-09-06T12:27:16.801 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:16 +0000 (0:00:00.019) 0:02:03.194 ****** 2021-09-06T12:27:17.452 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-09-06T12:27:17.454 INFO:teuthology.task.ansible.out:fatal: [smithi041.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-06T12:27:17.478 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:17 +0000 (0:00:00.677) 0:02:03.871 ****** 2021-09-06T12:27:17.497 INFO:teuthology.task.ansible.out:fatal: [smithi052.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2021-09-06T12:27:17.520 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:17 +0000 (0:00:00.041) 0:02:03.912 ****** 2021-09-06T12:27:17.548 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-09-06T12:27:17.550 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-06T12:27:17.572 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:17 +0000 (0:00:00.051) 0:02:03.964 ****** 2021-09-06T12:27:17.590 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2021-09-06T12:27:17.614 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:17 +0000 (0:00:00.042) 0:02:04.006 ****** 2021-09-06T12:27:35.590 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-06T12:27:35.592 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:27:35.614 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:35 +0000 (0:00:18.000) 0:02:22.006 ****** 2021-09-06T12:27:35.696 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-06T12:27:35.697 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-06T12:27:35.717 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:35 +0000 (0:00:00.103) 0:02:22.110 ****** 2021-09-06T12:27:36.436 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-09-06T12:27:36.438 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.480324", "end": "2021-09-06 12:27:36.415107", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:27:35.934783", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-06T12:27:36.458 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:36 +0000 (0:00:00.741) 0:02:22.851 ****** 2021-09-06T12:27:36.526 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-09-06T12:27:36.528 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:36.548 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:36 +0000 (0:00:00.090) 0:02:22.941 ****** 2021-09-06T12:27:36.617 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-09-06T12:27:36.619 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:36.640 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:36 +0000 (0:00:00.091) 0:02:23.032 ****** 2021-09-06T12:27:36.709 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-06T12:27:36.711 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T12:27:36.733 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:36 +0000 (0:00:00.092) 0:02:23.125 ****** 2021-09-06T12:27:36.802 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-06T12:27:36.804 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:36.823 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:36 +0000 (0:00:00.090) 0:02:23.216 ****** 2021-09-06T12:27:37.340 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-09-06T12:27:37.342 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:27:37.363 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:37 +0000 (0:00:00.539) 0:02:23.756 ****** 2021-09-06T12:27:37.443 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-09-06T12:27:37.444 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2021-09-06T12:27:37.464 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:37 +0000 (0:00:00.101) 0:02:23.857 ****** 2021-09-06T12:27:38.180 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.470900", "end": "2021-09-06 12:27:38.149721", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:27:37.678821", "stderr": "", "stderr_lines": [], "stdout": "7.8\n7.9\n7Server\n8\n8.2\n8.3\n8.4", "stdout_lines": ["7.8", "7.9", "7Server", "8", "8.2", "8.3", "8.4"]} 2021-09-06T12:27:38.202 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:38 +0000 (0:00:00.737) 0:02:24.594 ****** 2021-09-06T12:27:38.272 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-09-06T12:27:38.274 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:38.295 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:38 +0000 (0:00:00.093) 0:02:24.687 ****** 2021-09-06T12:27:38.365 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-09-06T12:27:38.367 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:38.387 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:38 +0000 (0:00:00.092) 0:02:24.780 ****** 2021-09-06T12:27:38.446 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-09-06T12:27:38.448 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-09-06T12:27:38.469 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:38 +0000 (0:00:00.081) 0:02:24.861 ****** 2021-09-06T12:27:38.549 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-06T12:27:38.551 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:38.572 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:38 +0000 (0:00:00.103) 0:02:24.965 ****** 2021-09-06T12:27:39.836 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-09-06T12:27:39.837 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.774493", "end": "2021-09-06 12:27:39.811281", "rc": 0, "start": "2021-09-06 12:27:37.036788", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-06T12:27:39.858 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:39 +0000 (0:00:01.285) 0:02:26.250 ****** 2021-09-06T12:27:39.937 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-06T12:27:39.939 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-06T12:27:39.958 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:39 +0000 (0:00:00.100) 0:02:26.350 ****** 2021-09-06T12:27:40.025 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-09-06T12:27:40.027 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:40.047 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:40 +0000 (0:00:00.088) 0:02:26.439 ****** 2021-09-06T12:27:40.125 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-06T12:27:40.127 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-06T12:27:40.145 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:40 +0000 (0:00:00.098) 0:02:26.538 ****** 2021-09-06T12:27:40.214 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-06T12:27:40.216 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-06T12:27:40.237 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:40 +0000 (0:00:00.091) 0:02:26.629 ****** 2021-09-06T12:27:41.564 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:02.733957", "end": "2021-09-06 12:27:41.545297", "rc": 0, "start": "2021-09-06 12:27:38.811340", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2021-09-06T12:27:41.587 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:41 +0000 (0:00:01.349) 0:02:27.979 ****** 2021-09-06T12:27:41.669 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-09-06T12:27:41.670 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2021-09-06T12:27:41.705 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:41 +0000 (0:00:00.117) 0:02:28.097 ****** 2021-09-06T12:27:41.777 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:41.800 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:41 +0000 (0:00:00.095) 0:02:28.193 ****** 2021-09-06T12:27:41.870 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-09-06T12:27:41.871 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2021-09-06T12:27:41.893 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:41 +0000 (0:00:00.092) 0:02:28.285 ****** 2021-09-06T12:27:41.962 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-09-06T12:27:41.964 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2021-09-06T12:27:41.985 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:41 +0000 (0:00:00.092) 0:02:28.377 ****** 2021-09-06T12:27:42.493 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-09-06T12:27:42.495 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:27:42.517 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:42 +0000 (0:00:00.531) 0:02:28.909 ****** 2021-09-06T12:27:44.243 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:27:44.265 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:44 +0000 (0:00:01.748) 0:02:30.657 ****** 2021-09-06T12:27:46.227 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-09-06T12:27:46.229 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.465801", "end": "2021-09-06 12:27:46.204432", "rc": 0, "start": "2021-09-06 12:27:42.738631", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-06T12:27:46.250 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:46 +0000 (0:00:01.985) 0:02:32.643 ****** 2021-09-06T12:27:46.496 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-06T12:27:46.497 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-06T12:27:46.518 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:46 +0000 (0:00:00.267) 0:02:32.910 ****** 2021-09-06T12:27:46.764 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-06T12:27:46.765 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:27:46.785 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:46 +0000 (0:00:00.266) 0:02:33.177 ****** 2021-09-06T12:27:47.299 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-06T12:27:47.301 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-06T12:27:47.758 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-06T12:27:47.782 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:47 +0000 (0:00:00.997) 0:02:34.174 ****** 2021-09-06T12:27:47.996 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-09-06T12:27:47.998 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["subscription-manager", "repos", "--enable=rhel-8-for-x86_64-baseos-rpms", "--enable=rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:03.490266", "end": "2021-09-06 12:27:47.973948", "rc": 0, "start": "2021-09-06 12:27:44.483682", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2021-09-06T12:27:48.022 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.239) 0:02:34.414 ****** 2021-09-06T12:27:48.256 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-09-06T12:27:48.258 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2021-09-06T12:27:48.279 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.257) 0:02:34.671 ****** 2021-09-06T12:27:48.316 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-06T12:27:48.318 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-06T12:27:48.515 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-09-06T12:27:48.517 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:27:48.536 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.257) 0:02:34.929 ****** 2021-09-06T12:27:48.771 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-06T12:27:48.773 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-06T12:27:48.795 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.258) 0:02:35.188 ****** 2021-09-06T12:27:48.874 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-06T12:27:48.876 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:48.896 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.100) 0:02:35.289 ****** 2021-09-06T12:27:48.964 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-06T12:27:48.966 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:48.986 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:48 +0000 (0:00:00.089) 0:02:35.378 ****** 2021-09-06T12:27:49.044 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-06T12:27:49.046 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.066 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.080) 0:02:35.459 ****** 2021-09-06T12:27:49.074 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-06T12:27:49.076 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-09-06T12:27:49.138 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.160 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.093) 0:02:35.552 ****** 2021-09-06T12:27:49.230 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-06T12:27:49.231 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.241 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.243 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.265 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.104) 0:02:35.657 ****** 2021-09-06T12:27:49.334 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-06T12:27:49.336 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.356 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.091) 0:02:35.749 ****** 2021-09-06T12:27:49.426 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-06T12:27:49.428 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.448 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.092) 0:02:35.841 ****** 2021-09-06T12:27:49.508 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-09-06T12:27:49.510 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-09-06T12:27:49.533 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.084) 0:02:35.926 ****** 2021-09-06T12:27:49.541 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-06T12:27:49.543 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.567 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.033) 0:02:35.959 ****** 2021-09-06T12:27:49.638 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-06T12:27:49.640 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.665 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.098) 0:02:36.057 ****** 2021-09-06T12:27:49.735 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-06T12:27:49.737 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:49.758 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:49 +0000 (0:00:00.092) 0:02:36.150 ****** 2021-09-06T12:27:50.070 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-09-06T12:27:50.072 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-09-06T12:27:50.524 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-06T12:27:50.526 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.565575", "end": "2021-09-06 12:27:50.500316", "rc": 0, "start": "2021-09-06 12:27:49.934741", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi052.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-09-06T12:27:50.560 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.802) 0:02:36.952 ****** 2021-09-06T12:27:50.592 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.032) 0:02:36.985 ****** 2021-09-06T12:27:50.673 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-09-06T12:27:50.674 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:50.696 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.103) 0:02:37.089 ****** 2021-09-06T12:27:50.765 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-06T12:27:50.767 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:50.786 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.090) 0:02:37.179 ****** 2021-09-06T12:27:50.853 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-09-06T12:27:50.855 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:50.874 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.087) 0:02:37.267 ****** 2021-09-06T12:27:50.941 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:50.963 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:50 +0000 (0:00:00.088) 0:02:37.355 ****** 2021-09-06T12:27:51.029 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-09-06T12:27:51.031 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.041 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.043 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.063 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.100) 0:02:37.456 ****** 2021-09-06T12:27:51.130 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-09-06T12:27:51.132 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.152 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.088) 0:02:37.545 ****** 2021-09-06T12:27:51.220 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-09-06T12:27:51.222 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.243 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.091) 0:02:37.636 ****** 2021-09-06T12:27:51.312 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-09-06T12:27:51.314 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.337 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.093) 0:02:37.730 ****** 2021-09-06T12:27:51.405 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-09-06T12:27:51.407 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.426 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.088) 0:02:37.818 ****** 2021-09-06T12:27:51.493 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-09-06T12:27:51.495 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:27:51.514 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:51 +0000 (0:00:00.088) 0:02:37.907 ****** 2021-09-06T12:27:52.526 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-09-06T12:27:52.527 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.808448", "end": "2021-09-06 12:27:52.500325", "rc": 0, "start": "2021-09-06 12:27:51.691877", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:27:52.547 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:27:52 +0000 (0:00:01.032) 0:02:38.940 ****** 2021-09-06T12:28:05.349 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-06T12:28:05.351 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:05.373 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:05 +0000 (0:00:12.825) 0:02:51.765 ****** 2021-09-06T12:28:05.454 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-06T12:28:05.455 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:05.476 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:05 +0000 (0:00:00.103) 0:02:51.869 ****** 2021-09-06T12:28:05.545 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-09-06T12:28:05.547 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:05.568 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:05 +0000 (0:00:00.091) 0:02:51.960 ****** 2021-09-06T12:28:05.638 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-06T12:28:05.639 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:05.661 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:05 +0000 (0:00:00.092) 0:02:52.053 ****** 2021-09-06T12:28:05.741 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-06T12:28:05.743 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:05.764 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:05 +0000 (0:00:00.103) 0:02:52.157 ****** 2021-09-06T12:28:06.267 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-09-06T12:28:06.269 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-06T12:28:06.298 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:06 +0000 (0:00:00.533) 0:02:52.691 ****** 2021-09-06T12:28:07.245 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-09-06T12:28:07.247 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:07.269 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:07 +0000 (0:00:00.970) 0:02:53.661 ****** 2021-09-06T12:28:07.338 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-09-06T12:28:07.340 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:07.362 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:07 +0000 (0:00:00.092) 0:02:53.754 ****** 2021-09-06T12:28:07.432 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:07.455 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:07 +0000 (0:00:00.093) 0:02:53.848 ****** 2021-09-06T12:28:07.525 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-09-06T12:28:07.526 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:07.548 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:07 +0000 (0:00:00.092) 0:02:53.940 ****** 2021-09-06T12:28:07.616 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-09-06T12:28:07.618 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:07.641 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:07 +0000 (0:00:00.093) 0:02:54.034 ****** 2021-09-06T12:28:08.172 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-09-06T12:28:08.174 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-09-06T12:28:08.199 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:08 +0000 (0:00:00.557) 0:02:54.591 ****** 2021-09-06T12:28:09.350 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-06T12:28:09.352 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:09.374 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:09 +0000 (0:00:01.175) 0:02:55.766 ****** 2021-09-06T12:28:09.904 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-06T12:28:09.906 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-06T12:28:10.352 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-06T12:28:10.796 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-06T12:28:10.822 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:10 +0000 (0:00:01.447) 0:02:57.214 ****** 2021-09-06T12:28:11.020 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-06T12:28:11.022 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:28:11.044 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:11 +0000 (0:00:00.222) 0:02:57.436 ****** 2021-09-06T12:28:11.356 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-09-06T12:28:11.358 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:11.380 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:11 +0000 (0:00:00.335) 0:02:57.772 ****** 2021-09-06T12:28:11.556 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-06T12:28:11.557 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-06T12:28:11.923 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-06T12:28:11.925 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-09-06T12:28:11.987 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-06T12:28:11.989 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-06T12:28:12.374 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-06T12:28:12.376 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-09-06T12:28:12.427 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-06T12:28:12.428 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-06T12:28:12.463 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:12 +0000 (0:00:01.083) 0:02:58.855 ****** 2021-09-06T12:28:12.532 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-06T12:28:12.534 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:12.554 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:12 +0000 (0:00:00.090) 0:02:58.946 ****** 2021-09-06T12:28:12.632 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-06T12:28:12.635 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-06T12:28:12.654 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:12 +0000 (0:00:00.100) 0:02:59.046 ****** 2021-09-06T12:28:12.816 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-09-06T12:28:12.818 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-09-06T12:28:12.842 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:12 +0000 (0:00:00.187) 0:02:59.234 ****** 2021-09-06T12:28:12.899 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-06T12:28:12.901 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:28:12.923 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:12 +0000 (0:00:00.081) 0:02:59.315 ****** 2021-09-06T12:28:13.012 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-06T12:28:13.014 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:13.023 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:13.035 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-09-06T12:28:13.036 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:28:13.057 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:13 +0000 (0:00:00.134) 0:02:59.449 ****** 2021-09-06T12:28:13.064 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-06T12:28:13.066 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:13.090 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:13 +0000 (0:00:00.032) 0:02:59.482 ****** 2021-09-06T12:28:13.575 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-09-06T12:28:13.577 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-09-06T12:28:14.012 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-09-06T12:28:14.458 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-09-06T12:28:14.492 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:14 +0000 (0:00:01.402) 0:03:00.885 ****** 2021-09-06T12:28:14.563 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-09-06T12:28:14.565 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:14.586 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:14 +0000 (0:00:00.094) 0:03:00.979 ****** 2021-09-06T12:28:14.666 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-09-06T12:28:14.668 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-09-06T12:28:14.690 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:14 +0000 (0:00:00.103) 0:03:01.082 ****** 2021-09-06T12:28:14.949 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-09-06T12:28:14.951 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-09-06T12:28:14.975 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:14 +0000 (0:00:00.285) 0:03:01.368 ****** 2021-09-06T12:28:15.056 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-09-06T12:28:15.057 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:15.078 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:15.089 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:15.100 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:15.123 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:15 +0000 (0:00:00.147) 0:03:01.515 ****** 2021-09-06T12:28:16.472 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-06T12:28:16.474 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:16.498 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:16 +0000 (0:00:01.375) 0:03:02.890 ****** 2021-09-06T12:28:16.569 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-06T12:28:16.571 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:16.594 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:16 +0000 (0:00:00.095) 0:03:02.986 ****** 2021-09-06T12:28:17.115 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-06T12:28:17.117 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-06T12:28:17.138 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:17 +0000 (0:00:00.544) 0:03:03.530 ****** 2021-09-06T12:28:17.637 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-06T12:28:17.638 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-06T12:28:17.659 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:17 +0000 (0:00:00.521) 0:03:04.052 ****** 2021-09-06T12:28:17.771 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-06T12:28:17.773 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:17.793 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:17.818 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:17 +0000 (0:00:00.158) 0:03:04.211 ****** 2021-09-06T12:28:17.820 INFO:teuthology.task.ansible.out: 2021-09-06T12:28:18.072 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-06T12:28:18.074 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:28:18.096 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:18 +0000 (0:00:00.277) 0:03:04.488 ****** 2021-09-06T12:28:18.342 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-09-06T12:28:18.344 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:18.365 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:18 +0000 (0:00:00.269) 0:03:04.758 ****** 2021-09-06T12:28:18.372 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-06T12:28:18.374 INFO:teuthology.task.ansible.out:fatal: [smithi041.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-06T12:28:18.397 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:18 +0000 (0:00:00.031) 0:03:04.789 ****** 2021-09-06T12:28:18.446 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-09-06T12:28:18.448 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:18.470 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:18 +0000 (0:00:00.073) 0:03:04.863 ****** 2021-09-06T12:28:18.490 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-09-06T12:28:18.491 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:18.525 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:18 +0000 (0:00:00.054) 0:03:04.917 ****** 2021-09-06T12:28:19.004 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-09-06T12:28:19.006 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-09-06T12:28:19.028 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.503) 0:03:05.420 ****** 2021-09-06T12:28:19.046 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-06T12:28:19.048 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-06T12:28:19.072 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.044) 0:03:05.465 ****** 2021-09-06T12:28:19.563 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-09-06T12:28:19.565 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-09-06T12:28:19.589 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.516) 0:03:05.981 ****** 2021-09-06T12:28:19.681 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-09-06T12:28:19.683 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:19.693 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-09-06T12:28:19.720 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.131) 0:03:06.112 ****** 2021-09-06T12:28:19.760 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-06T12:28:19.762 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-09-06 12:24:21 UTC", "ActiveEnterTimestampMonotonic": "25092135", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target time-sync.target var-run.mount local-fs.target basic.target systemd-journald.socket sysinit.target system.slice tmp.mount -.mount network-online.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-09-06 12:24:21 UTC", "AssertTimestampMonotonic": "25077977", "Before": "shutdown.target multi-user.target getty@tty1.service plymouth-quit.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-09-06 12:24:21 UTC", "ConditionT 2021-09-06T12:28:19.764 INFO:teuthology.task.ansible.out:imestampMonotonic": "25077977", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1233", "ExecMainStartTimestamp": "Mon 2021-09-06 12:24:21 UTC", "ExecMainStartTimestampMonotonic": "25091680", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Mon 2021-09-06 12:24:21 UTC] ; stop_time=[n/a] ; pid=1233 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-09-06 12:24:21 UTC", "InactiveExitTimestampMonotonic": "25092135", "In 2021-09-06T12:28:19.766 INFO:teuthology.task.ansible.out:vocationID": "738affc4692847eb949c5c8438445553", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1233", "MemoryAccounting": "yes", "MemoryCurrent": "667648", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-09-06T12:28:19.768 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-09-06 12:24:21 UTC", "StateChangeTimestampMonotonic": "25092135", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-09-06 12:24:21 UTC", "WatchdogTimestampMonotonic": "25092115", "WatchdogUSec": "0"}} 2021-09-06T12:28:19.785 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.065) 0:03:06.178 ****** 2021-09-06T12:28:19.968 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-09-06T12:28:19.970 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:28:19.991 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:19 +0000 (0:00:00.205) 0:03:06.383 ****** 2021-09-06T12:28:20.029 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-06T12:28:20.031 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001857", "end": "2021-09-06 12:28:20.007352", "rc": 0, "start": "2021-09-06 12:28:20.005495", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-06T12:28:20.071 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:20 +0000 (0:00:00.080) 0:03:06.464 ****** 2021-09-06T12:28:20.280 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-09-06T12:28:20.282 INFO:teuthology.task.ansible.out:fatal: [smithi052.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-09-06T12:28:20.305 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:20 +0000 (0:00:00.234) 0:03:06.698 ****** 2021-09-06T12:28:20.386 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:28:20.411 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:20 +0000 (0:00:00.105) 0:03:06.803 ****** 2021-09-06T12:28:20.927 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-09-06T12:28:20.929 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-09-06T12:28:20.950 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:20 +0000 (0:00:00.539) 0:03:07.342 ****** 2021-09-06T12:28:21.309 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-09-06T12:28:21.310 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-09-06 12:24:15 UTC", "ActiveEnterTimestampMonotonic": "25576731", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount tmp.mount sysinit.target systemd-tmpfiles-setup.service basic.target -.mount network-online.target local-fs.target nss-lookup.target time-sync.target systemd-journald.socket system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-09-06 12:24:15 UTC", "AssertTimestampMonotonic": "25549311", "Before": "multi-user.target getty@tty1.service shutdown.target plymouth-quit.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-09-06 12:24:15 UTC", "ConditionT 2021-09-06T12:28:21.312 INFO:teuthology.task.ansible.out:imestampMonotonic": "25549310", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1226", "ExecMainStartTimestamp": "Mon 2021-09-06 12:24:15 UTC", "ExecMainStartTimestampMonotonic": "25576704", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Mon 2021-09-06 12:24:15 UTC] ; stop_time=[n/a] ; pid=1226 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "FreezerState": "running", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-09-06 12:24:15 UTC", "InactiveExitTimestampMonotonic": "25576731", "In 2021-09-06T12:28:21.313 INFO:teuthology.task.ansible.out:vocationID": "341f85e5ca7e44bcb920317ad3bfef0d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "1226", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGr 2021-09-06T12:28:21.315 INFO:teuthology.task.ansible.out:oups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-09-06 12:24:15 UTC", "StateChangeTimestampMonotonic": "25576731", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-09-06 12:24:15 UTC", "WatchdogTimestampMonotonic": "25576730", "WatchdogUSec": "0"}} 2021-09-06T12:28:21.333 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:21 +0000 (0:00:00.383) 0:03:07.726 ****** 2021-09-06T12:28:21.590 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-09-06T12:28:21.592 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001965", "end": "2021-09-06 12:28:21.567446", "rc": 0, "start": "2021-09-06 12:28:21.565481", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-09-06T12:28:21.612 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:21 +0000 (0:00:00.279) 0:03:08.005 ****** 2021-09-06T12:28:23.241 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-09-06T12:28:23.243 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:23.269 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:23 +0000 (0:00:01.656) 0:03:09.661 ****** 2021-09-06T12:28:24.797 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:28:24.821 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:28:24 +0000 (0:00:01.552) 0:03:11.213 ****** 2021-09-06T12:29:17.377 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-06T12:29:17.379 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-06T12:29:17.402 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:17 +0000 (0:00:52.581) 0:04:03.795 ****** 2021-09-06T12:29:18.159 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-06T12:29:18.161 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-06T12:29:18.181 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:18 +0000 (0:00:00.778) 0:04:04.573 ****** 2021-09-06T12:29:19.040 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-09-06T12:29:19.042 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch"]} 2021-09-06T12:29:19.063 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:19 +0000 (0:00:00.882) 0:04:05.456 ****** 2021-09-06T12:29:19.571 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-09-06T12:29:19.573 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-09-06T12:29:19.594 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:19 +0000 (0:00:00.530) 0:04:05.986 ****** 2021-09-06T12:29:34.233 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-06T12:29:34.235 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.808907", "end": "2021-09-06 12:29:34.208764", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:29:18.399857", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-06T12:29:34.256 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:34 +0000 (0:00:14.662) 0:04:20.649 ****** 2021-09-06T12:29:34.775 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-06T12:29:34.776 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-06T12:29:34.797 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:34 +0000 (0:00:00.540) 0:04:21.189 ****** 2021-09-06T12:29:35.040 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-06T12:29:35.042 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017031", "end": "2021-09-06 12:29:35.020248", "rc": 0, "start": "2021-09-06 12:29:35.003217", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:35.062 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:35 +0000 (0:00:00.265) 0:04:21.454 ****** 2021-09-06T12:29:35.296 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-06T12:29:35.298 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012726", "end": "2021-09-06 12:29:35.281761", "rc": 0, "start": "2021-09-06 12:29:35.269035", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:35.320 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:35 +0000 (0:00:00.257) 0:04:21.712 ****** 2021-09-06T12:29:35.753 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-09-06T12:29:35.755 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:15.922031", "end": "2021-09-06 12:29:35.726869", "failed_when_result": false, "rc": 0, "start": "2021-09-06 12:29:19.804838", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-09-06T12:29:35.776 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:35 +0000 (0:00:00.456) 0:04:22.168 ****** 2021-09-06T12:29:36.291 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-09-06T12:29:36.291 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-09-06T12:29:36.312 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:36 +0000 (0:00:00.535) 0:04:22.704 ****** 2021-09-06T12:29:36.560 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-09-06T12:29:36.562 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013302", "end": "2021-09-06 12:29:36.538787", "rc": 0, "start": "2021-09-06 12:29:36.525485", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:36.582 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:36 +0000 (0:00:00.269) 0:04:22.974 ****** 2021-09-06T12:29:36.839 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-09-06T12:29:36.841 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.016262", "end": "2021-09-06 12:29:36.816744", "rc": 0, "start": "2021-09-06 12:29:36.800482", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:36.862 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:36 +0000 (0:00:00.280) 0:04:23.254 ****** 2021-09-06T12:29:51.232 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-06T12:29:51.234 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.678837", "end": "2021-09-06 12:29:51.211902", "rc": 0, "start": "2021-09-06 12:29:35.533065", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:51.256 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:14.393) 0:04:37.648 ****** 2021-09-06T12:29:51.490 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-06T12:29:51.492 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-06T12:29:51.512 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:00.256) 0:04:37.905 ****** 2021-09-06T12:29:51.581 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-06T12:29:51.582 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-06T12:29:51.603 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:00.090) 0:04:37.996 ****** 2021-09-06T12:29:51.682 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-06T12:29:51.684 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T12:29:51.704 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:00.101) 0:04:38.097 ****** 2021-09-06T12:29:51.783 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-06T12:29:51.785 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-06T12:29:51.806 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:00.102) 0:04:38.199 ****** 2021-09-06T12:29:51.875 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-06T12:29:51.877 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T12:29:51.897 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:51 +0000 (0:00:00.090) 0:04:38.289 ****** 2021-09-06T12:29:52.090 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-09-06T12:29:52.091 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T12:29:52.111 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.214) 0:04:38.503 ****** 2021-09-06T12:29:52.345 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-06T12:29:52.347 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-06T12:29:52.367 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.255) 0:04:38.759 ****** 2021-09-06T12:29:52.425 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-06T12:29:52.426 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:29:52.448 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.081) 0:04:38.841 ****** 2021-09-06T12:29:52.506 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-06T12:29:52.508 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:29:52.528 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.080) 0:04:38.921 ****** 2021-09-06T12:29:52.794 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-06T12:29:52.796 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-06T12:29:52.816 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.287) 0:04:39.209 ****** 2021-09-06T12:29:52.844 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-09-06T12:29:52.846 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.756750", "end": "2021-09-06 12:29:52.826571", "rc": 0, "start": "2021-09-06 12:29:37.069821", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:29:52.868 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:52 +0000 (0:00:00.051) 0:04:39.260 ****** 2021-09-06T12:29:53.104 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-09-06T12:29:53.105 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-09-06T12:29:53.125 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.257) 0:04:39.518 ****** 2021-09-06T12:29:53.194 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-09-06T12:29:53.196 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-09-06T12:29:53.216 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.090) 0:04:39.609 ****** 2021-09-06T12:29:53.285 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-09-06T12:29:53.287 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T12:29:53.308 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.091) 0:04:39.701 ****** 2021-09-06T12:29:53.376 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-09-06T12:29:53.378 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "redhat-lsb", "firewalld", "wget", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "make", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb", "iozone", "lvm2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2021-09-06T12:29:53.400 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.091) 0:04:39.792 ****** 2021-09-06T12:29:53.406 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-06T12:29:53.407 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-06T12:29:53.430 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.030) 0:04:39.823 ****** 2021-09-06T12:29:53.469 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-09-06T12:29:53.470 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-09-06T12:29:53.493 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.062) 0:04:39.885 ****** 2021-09-06T12:29:53.509 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-06T12:29:53.511 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi041"}, "changed": false} 2021-09-06T12:29:53.532 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.039) 0:04:39.925 ****** 2021-09-06T12:29:53.685 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-09-06T12:29:53.687 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-09-06T12:29:53.707 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.174) 0:04:40.099 ****** 2021-09-06T12:29:53.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-09-06T12:29:53.955 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-09-06T12:29:53.976 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:53 +0000 (0:00:00.268) 0:04:40.368 ****** 2021-09-06T12:29:54.045 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-09-06T12:29:54.047 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:29:54.067 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:54 +0000 (0:00:00.091) 0:04:40.460 ****** 2021-09-06T12:29:54.136 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-09-06T12:29:54.138 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:29:54.162 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:54 +0000 (0:00:00.094) 0:04:40.554 ****** 2021-09-06T12:29:54.335 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-09-06T12:29:54.336 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi041", "ansible_hostname": "smithi041", "ansible_nodename": "smithi041"}, "changed": false, "name": "smithi041"} 2021-09-06T12:29:54.358 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:54 +0000 (0:00:00.196) 0:04:40.750 ****** 2021-09-06T12:29:54.428 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-09-06T12:29:54.429 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-09-06T12:29:54.452 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:54 +0000 (0:00:00.094) 0:04:40.844 ****** 2021-09-06T12:29:54.852 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-09-06T12:29:54.854 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-06T12:29:54.874 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:54 +0000 (0:00:00.422) 0:04:41.267 ****** 2021-09-06T12:29:55.026 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-09-06T12:29:55.028 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-09-06T12:29:55.049 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:55 +0000 (0:00:00.174) 0:04:41.441 ****** 2021-09-06T12:29:55.116 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-09-06T12:29:55.118 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi052"}, "changed": false} 2021-09-06T12:29:55.137 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:55 +0000 (0:00:00.088) 0:04:41.530 ****** 2021-09-06T12:29:55.495 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-09-06T12:29:55.497 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-06T12:29:55.517 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:55 +0000 (0:00:00.379) 0:04:41.909 ****** 2021-09-06T12:29:55.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-09-06T12:29:55.618 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi052", "ansible_hostname": "smithi052", "ansible_nodename": "smithi052"}, "changed": false, "name": "smithi052"} 2021-09-06T12:29:55.638 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:55 +0000 (0:00:00.121) 0:04:42.030 ****** 2021-09-06T12:29:56.140 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-09-06T12:29:56.142 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-09-06T12:29:56.162 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:56 +0000 (0:00:00.523) 0:04:42.554 ****** 2021-09-06T12:29:56.168 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-09-06T12:29:56.168 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-06T12:29:56.192 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:56 +0000 (0:00:00.030) 0:04:42.584 ****** 2021-09-06T12:29:56.680 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-09-06T12:29:56.682 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-09-06T12:29:56.703 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:56 +0000 (0:00:00.511) 0:04:43.096 ****** 2021-09-06T12:29:56.711 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-06T12:29:56.713 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-06T12:29:56.736 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:56 +0000 (0:00:00.032) 0:04:43.128 ****** 2021-09-06T12:29:56.971 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T12:29:56.972 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T12:29:56.992 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:56 +0000 (0:00:00.255) 0:04:43.384 ****** 2021-09-06T12:29:57.215 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T12:29:57.217 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-06T12:29:57.236 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:57 +0000 (0:00:00.244) 0:04:43.629 ****** 2021-09-06T12:29:57.377 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-09-06T12:29:57.379 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-09-06T12:29:57.399 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:57 +0000 (0:00:00.163) 0:04:43.792 ****** 2021-09-06T12:29:57.509 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T12:29:57.510 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-06T12:29:57.531 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:57 +0000 (0:00:00.131) 0:04:43.924 ****** 2021-09-06T12:29:57.909 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-09-06T12:29:57.911 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-09-06T12:29:57.931 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:57 +0000 (0:00:00.400) 0:04:44.324 ****** 2021-09-06T12:29:58.041 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-06T12:29:58.042 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-06T12:29:58.063 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:58 +0000 (0:00:00.131) 0:04:44.455 ****** 2021-09-06T12:29:58.163 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T12:29:58.165 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T12:29:58.185 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:58 +0000 (0:00:00.121) 0:04:44.577 ****** 2021-09-06T12:29:58.417 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T12:29:58.419 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-09-06T12:29:58.437 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:58 +0000 (0:00:00.252) 0:04:44.830 ****** 2021-09-06T12:29:58.576 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T12:29:58.578 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-06T12:29:58.601 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:58 +0000 (0:00:00.163) 0:04:44.993 ****** 2021-09-06T12:29:58.711 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T12:29:58.712 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-09-06T12:29:58.734 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:58 +0000 (0:00:00.132) 0:04:45.126 ****** 2021-09-06T12:29:59.248 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-06T12:29:59.249 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-09-06T12:29:59.269 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:59 +0000 (0:00:00.535) 0:04:45.662 ****** 2021-09-06T12:29:59.773 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T12:29:59.775 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-09-06T12:29:59.795 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:29:59 +0000 (0:00:00.525) 0:04:46.187 ****** 2021-09-06T12:30:01.130 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T12:30:01.131 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.294811", "end": "2021-09-06 12:30:01.107053", "rc": 0, "start": "2021-09-06 12:29:58.812242", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:30:01.153 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:01 +0000 (0:00:01.357) 0:04:47.545 ****** 2021-09-06T12:30:01.222 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-06T12:30:01.222 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:01.242 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:01 +0000 (0:00:00.089) 0:04:47.635 ****** 2021-09-06T12:30:01.310 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:01.332 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:01 +0000 (0:00:00.089) 0:04:47.724 ****** 2021-09-06T12:30:01.421 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-06T12:30:01.422 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-06T12:30:01.445 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:01 +0000 (0:00:00.112) 0:04:47.837 ****** 2021-09-06T12:30:01.512 INFO:teuthology.task.ansible.out: 2021-09-06T12:30:01.514 INFO:teuthology.task.ansible.out:TASK [testnode : Configure local mirrorlists] ********************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:01.534 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:01 +0000 (0:00:00.089) 0:04:47.926 ****** 2021-09-06T12:30:02.070 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-06T12:30:02.072 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-06T12:30:02.092 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.558) 0:04:48.485 ****** 2021-09-06T12:30:02.150 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-06T12:30:02.172 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.079) 0:04:48.564 ****** 2021-09-06T12:30:02.191 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.162391", "end": "2021-09-06 12:30:02.161081", "rc": 0, "start": "2021-09-06 12:29:59.998690", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:30:02.215 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.043) 0:04:48.608 ****** 2021-09-06T12:30:02.287 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ 2021-09-06T12:30:02.288 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:02.309 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.093) 0:04:48.702 ****** 2021-09-06T12:30:02.377 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:30:02.379 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:02.398 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.089) 0:04:48.791 ****** 2021-09-06T12:30:02.477 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* 2021-09-06T12:30:02.479 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-09-06T12:30:02.499 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.100) 0:04:48.891 ****** 2021-09-06T12:30:02.567 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-09-06T12:30:02.569 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:02.590 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:02 +0000 (0:00:00.091) 0:04:48.983 ****** 2021-09-06T12:30:03.143 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-09-06T12:30:03.145 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-09-06T12:30:03.167 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.576) 0:04:49.559 ****** 2021-09-06T12:30:03.206 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-06T12:30:03.208 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.797729", "end": "2021-09-06 12:30:03.185609", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-06 12:30:02.387880", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use 2021-09-06T12:30:03.209 INFO:teuthology.task.ansible.out: command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2021-09-06T12:30:03.233 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.066) 0:04:49.625 ****** 2021-09-06T12:30:03.250 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-09-06T12:30:03.273 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.040) 0:04:49.665 ****** 2021-09-06T12:30:03.301 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-06T12:30:03.303 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.326 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.052) 0:04:49.718 ****** 2021-09-06T12:30:03.407 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-06T12:30:03.409 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.430 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.103) 0:04:49.822 ****** 2021-09-06T12:30:03.499 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-06T12:30:03.501 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.519 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.089) 0:04:49.912 ****** 2021-09-06T12:30:03.589 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:30:03.591 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.611 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.091) 0:04:50.004 ****** 2021-09-06T12:30:03.681 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-06T12:30:03.683 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.704 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.092) 0:04:50.096 ****** 2021-09-06T12:30:03.773 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-06T12:30:03.774 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:03.794 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:03 +0000 (0:00:00.090) 0:04:50.187 ****** 2021-09-06T12:30:04.051 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-09-06T12:30:04.053 INFO:teuthology.task.ansible.out:fatal: [smithi041.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007105", "end": "2021-09-06 12:30:04.028248", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 12:30:04.021143", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-06T12:30:04.074 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.279) 0:04:50.467 ****** 2021-09-06T12:30:04.145 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-06T12:30:04.146 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.167 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.092) 0:04:50.559 ****** 2021-09-06T12:30:04.226 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-09-06T12:30:04.228 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.714447", "end": "2021-09-06 12:30:04.207599", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-09-06 12:30:03.493152", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2021-09-06T12:30:04.251 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.083) 0:04:50.643 ****** 2021-09-06T12:30:04.319 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-09-06T12:30:04.320 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.342 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.090) 0:04:50.734 ****** 2021-09-06T12:30:04.421 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-09-06T12:30:04.423 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.444 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.102) 0:04:50.836 ****** 2021-09-06T12:30:04.513 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-09-06T12:30:04.515 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.539 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.094) 0:04:50.931 ****** 2021-09-06T12:30:04.607 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:30:04.609 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.630 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.090) 0:04:51.022 ****** 2021-09-06T12:30:04.698 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-09-06T12:30:04.700 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.720 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.089) 0:04:51.112 ****** 2021-09-06T12:30:04.799 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-09-06T12:30:04.801 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:04.823 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:04 +0000 (0:00:00.103) 0:04:51.215 ****** 2021-09-06T12:30:05.070 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi052.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007113", "end": "2021-09-06 12:30:05.052090", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 12:30:05.044977", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-09-06T12:30:05.093 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:05 +0000 (0:00:00.270) 0:04:51.486 ****** 2021-09-06T12:30:05.172 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-09-06T12:30:05.174 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:30:05.193 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:05 +0000 (0:00:00.100) 0:04:51.586 ****** 2021-09-06T12:30:35.131 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-06T12:30:35.133 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-06T12:30:35.156 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:35 +0000 (0:00:29.962) 0:05:21.548 ****** 2021-09-06T12:30:36.025 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-06T12:30:36.048 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:30:36 +0000 (0:00:00.892) 0:05:22.441 ****** 2021-09-06T12:31:03.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-06T12:31:03.762 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-06T12:31:03.783 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:03 +0000 (0:00:27.734) 0:05:50.175 ****** 2021-09-06T12:31:05.574 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-09-06T12:31:05.598 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:05 +0000 (0:00:01.814) 0:05:51.990 ****** 2021-09-06T12:31:36.582 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T12:31:36.584 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-06T12:31:36.605 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:36 +0000 (0:00:31.007) 0:06:22.998 ****** 2021-09-06T12:31:38.715 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: qemu-img-15:4.2.0-48.module+el8.4.0+11909+3300d70f.3.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-img-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module+el8.4.0+10368+630e803b.x86_64"]} 2021-09-06T12:31:38.738 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:38 +0000 (0:00:02.132) 0:06:25.130 ****** 2021-09-06T12:31:50.826 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-09-06T12:31:50.827 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:31:50.848 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:50 +0000 (0:00:12.110) 0:06:37.241 ****** 2021-09-06T12:31:52.897 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:31:52.919 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:52 +0000 (0:00:02.070) 0:06:39.311 ****** 2021-09-06T12:31:54.011 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T12:31:54.013 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:31:54.034 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:54 +0000 (0:00:01.115) 0:06:40.427 ****** 2021-09-06T12:31:54.102 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:31:54.104 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:54.123 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:54 +0000 (0:00:00.088) 0:06:40.515 ****** 2021-09-06T12:31:54.469 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-06T12:31:54.470 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target basic.target dbus.socket polkit.service system.slice dbus.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target ipset.service ebtables.service nftables.service iptables.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-09-06T12:31:54.472 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-09-06T12:31:54.473 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-09-06T12:31:54.475 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-06T12:31:54.492 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:54 +0000 (0:00:00.368) 0:06:40.884 ****** 2021-09-06T12:31:54.818 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-06T12:31:54.820 INFO:teuthology.task.ansible.out:fatal: [smithi041.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-09-06T12:31:54.839 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:54 +0000 (0:00:00.347) 0:06:41.232 ****** 2021-09-06T12:31:55.563 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-09-06T12:31:55.564 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-06T12:31:55.584 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.744) 0:06:41.976 ****** 2021-09-06T12:31:55.641 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-06T12:31:55.643 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:55.662 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.077) 0:06:42.054 ****** 2021-09-06T12:31:55.723 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-06T12:31:55.725 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:55.744 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.082) 0:06:42.137 ****** 2021-09-06T12:31:55.802 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-06T12:31:55.803 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:55.822 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.078) 0:06:42.215 ****** 2021-09-06T12:31:55.880 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-06T12:31:55.881 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:55.901 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.078) 0:06:42.294 ****** 2021-09-06T12:31:55.959 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-06T12:31:55.961 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:55.980 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:55 +0000 (0:00:00.079) 0:06:42.373 ****** 2021-09-06T12:31:56.037 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-06T12:31:56.038 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.057 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.076) 0:06:42.449 ****** 2021-09-06T12:31:56.114 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-06T12:31:56.134 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.077) 0:06:42.526 ****** 2021-09-06T12:31:56.140 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T12:31:56.141 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:31:56.164 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.029) 0:06:42.556 ****** 2021-09-06T12:31:56.191 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-06T12:31:56.192 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.212 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.048) 0:06:42.605 ****** 2021-09-06T12:31:56.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:31:56.231 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.252 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.039) 0:06:42.645 ****** 2021-09-06T12:31:56.279 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-06T12:31:56.281 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.302 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.050) 0:06:42.695 ****** 2021-09-06T12:31:56.371 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-06T12:31:56.373 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.382 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.407 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.104) 0:06:42.799 ****** 2021-09-06T12:31:56.464 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-06T12:31:56.466 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.485 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.078) 0:06:42.878 ****** 2021-09-06T12:31:56.544 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-06T12:31:56.565 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.079) 0:06:42.957 ****** 2021-09-06T12:31:56.592 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-09-06T12:31:56.594 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service dbus.service sysinit.target system.slice basic.target dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nftables.service ebtables.service ip6tables.service ipset.service iptables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", 2021-09-06T12:31:56.595 INFO:teuthology.task.ansible.out:"Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": " 2021-09-06T12:31:56.597 INFO:teuthology.task.ansible.out:0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null 2021-09-06T12:31:56.599 INFO:teuthology.task.ansible.out:", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-09-06T12:31:56.617 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.052) 0:06:43.010 ****** 2021-09-06T12:31:56.633 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-06T12:31:56.635 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.655 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.038) 0:06:43.048 ****** 2021-09-06T12:31:56.713 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-06T12:31:56.715 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.735 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.079) 0:06:43.127 ****** 2021-09-06T12:31:56.791 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-06T12:31:56.793 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.812 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.077) 0:06:43.205 ****** 2021-09-06T12:31:56.868 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:31:56.870 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.889 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.076) 0:06:43.281 ****** 2021-09-06T12:31:56.945 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-09-06T12:31:56.947 INFO:teuthology.task.ansible.out:fatal: [smithi052.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:56.967 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.078) 0:06:43.360 ****** 2021-09-06T12:31:56.987 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:56 +0000 (0:00:00.019) 0:06:43.380 ****** 2021-09-06T12:31:57.025 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T12:31:57.027 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.048 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.061) 0:06:43.441 ****** 2021-09-06T12:31:57.117 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-06T12:31:57.119 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.138 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.090) 0:06:43.531 ****** 2021-09-06T12:31:57.207 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-06T12:31:57.209 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.228 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.089) 0:06:43.620 ****** 2021-09-06T12:31:57.295 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-06T12:31:57.297 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Enable SELinux] *********************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-09-06T12:31:57.318 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.090) 0:06:43.710 ****** 2021-09-06T12:31:57.336 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.017) 0:06:43.728 ****** 2021-09-06T12:31:57.384 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-06T12:31:57.386 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.410 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.073) 0:06:43.802 ****** 2021-09-06T12:31:57.416 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-09-06T12:31:57.417 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.439 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.029) 0:06:43.832 ****** 2021-09-06T12:31:57.467 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-09-06T12:31:57.469 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.490 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.050) 0:06:43.882 ****** 2021-09-06T12:31:57.507 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-09-06T12:31:57.509 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.530 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.039) 0:06:43.922 ****** 2021-09-06T12:31:57.546 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-06T12:31:57.548 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.570 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.039) 0:06:43.962 ****** 2021-09-06T12:31:57.597 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-09-06T12:31:57.599 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.620 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.050) 0:06:44.013 ****** 2021-09-06T12:31:57.638 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-06T12:31:57.640 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.661 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.040) 0:06:44.053 ****** 2021-09-06T12:31:57.691 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-09-06T12:31:57.693 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.713 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.052) 0:06:44.106 ****** 2021-09-06T12:31:57.730 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-06T12:31:57.731 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.753 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.039) 0:06:44.145 ****** 2021-09-06T12:31:57.771 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-09-06T12:31:57.772 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.794 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.041) 0:06:44.187 ****** 2021-09-06T12:31:57.821 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-06T12:31:57.822 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.844 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.050) 0:06:44.237 ****** 2021-09-06T12:31:57.862 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-09-06T12:31:57.863 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.885 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.040) 0:06:44.277 ****** 2021-09-06T12:31:57.902 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T12:31:57.903 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:57.925 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.040) 0:06:44.317 ****** 2021-09-06T12:31:57.942 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-09-06T12:31:57.965 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:57 +0000 (0:00:00.040) 0:06:44.358 ****** 2021-09-06T12:31:57.993 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T12:31:57.994 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.016 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.050) 0:06:44.408 ****** 2021-09-06T12:31:58.033 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-09-06T12:31:58.035 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.056 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:44.449 ****** 2021-09-06T12:31:58.252 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T12:31:58.254 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.274 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.217) 0:06:44.666 ****** 2021-09-06T12:31:58.291 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-09-06T12:31:58.293 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.315 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:44.707 ****** 2021-09-06T12:31:58.342 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-09-06T12:31:58.343 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.364 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.049) 0:06:44.757 ****** 2021-09-06T12:31:58.392 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-09-06T12:31:58.393 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.415 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.050) 0:06:44.807 ****** 2021-09-06T12:31:58.431 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T12:31:58.433 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.455 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.039) 0:06:44.847 ****** 2021-09-06T12:31:58.481 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-09-06T12:31:58.483 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.504 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.048) 0:06:44.896 ****** 2021-09-06T12:31:58.520 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T12:31:58.522 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.546 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.042) 0:06:44.938 ****** 2021-09-06T12:31:58.562 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-09-06T12:31:58.585 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.039) 0:06:44.977 ****** 2021-09-06T12:31:58.612 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-06T12:31:58.613 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.635 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.050) 0:06:45.028 ****** 2021-09-06T12:31:58.654 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-09-06T12:31:58.655 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.676 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:45.068 ****** 2021-09-06T12:31:58.692 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-06T12:31:58.694 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.715 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.039) 0:06:45.108 ****** 2021-09-06T12:31:58.742 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-09-06T12:31:58.744 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.766 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.050) 0:06:45.158 ****** 2021-09-06T12:31:58.783 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:31:58.785 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.806 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:45.198 ****** 2021-09-06T12:31:58.824 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-09-06T12:31:58.826 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.846 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:45.239 ****** 2021-09-06T12:31:58.873 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.897 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.050) 0:06:45.289 ****** 2021-09-06T12:31:58.914 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:31:58.916 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.937 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.040) 0:06:45.330 ****** 2021-09-06T12:31:58.953 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T12:31:58.955 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:58.976 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:58 +0000 (0:00:00.038) 0:06:45.369 ****** 2021-09-06T12:31:59.004 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-09-06T12:31:59.005 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.027 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.050) 0:06:45.419 ****** 2021-09-06T12:31:59.044 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:31:59.045 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.067 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.040) 0:06:45.459 ****** 2021-09-06T12:31:59.084 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T12:31:59.086 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.107 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.040) 0:06:45.500 ****** 2021-09-06T12:31:59.134 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T12:31:59.136 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.158 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.050) 0:06:45.550 ****** 2021-09-06T12:31:59.175 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-09-06T12:31:59.177 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.198 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.040) 0:06:45.591 ****** 2021-09-06T12:31:59.381 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:31:59.383 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.402 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.203) 0:06:45.794 ****** 2021-09-06T12:31:59.419 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-09-06T12:31:59.421 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.442 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.040) 0:06:45.835 ****** 2021-09-06T12:31:59.616 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:31:59.618 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.638 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.195) 0:06:46.030 ****** 2021-09-06T12:31:59.655 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-09-06T12:31:59.656 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.677 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.039) 0:06:46.070 ****** 2021-09-06T12:31:59.865 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:31:59.867 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.891 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.213) 0:06:46.283 ****** 2021-09-06T12:31:59.898 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-09-06T12:31:59.900 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:31:59.923 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:31:59 +0000 (0:00:00.031) 0:06:46.315 ****** 2021-09-06T12:32:00.222 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T12:32:00.224 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.242 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.319) 0:06:46.635 ****** 2021-09-06T12:32:00.262 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.018) 0:06:46.654 ****** 2021-09-06T12:32:00.309 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T12:32:00.310 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.330 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.068) 0:06:46.723 ****** 2021-09-06T12:32:00.336 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-09-06T12:32:00.338 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.359 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.028) 0:06:46.752 ****** 2021-09-06T12:32:00.396 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T12:32:00.397 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.419 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.059) 0:06:46.812 ****** 2021-09-06T12:32:00.435 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-09-06T12:32:00.437 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.457 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.038) 0:06:46.850 ****** 2021-09-06T12:32:00.484 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:32:00.485 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.506 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.048) 0:06:46.899 ****** 2021-09-06T12:32:00.524 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-09-06T12:32:00.525 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.546 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.039) 0:06:46.938 ****** 2021-09-06T12:32:00.572 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:32:00.574 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.597 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.051) 0:06:46.990 ****** 2021-09-06T12:32:00.616 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-09-06T12:32:00.617 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.640 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.042) 0:06:47.032 ****** 2021-09-06T12:32:00.668 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:32:00.670 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.691 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.051) 0:06:47.083 ****** 2021-09-06T12:32:00.708 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-09-06T12:32:00.710 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.733 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.041) 0:06:47.125 ****** 2021-09-06T12:32:00.761 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:32:00.762 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.785 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.052) 0:06:47.177 ****** 2021-09-06T12:32:00.803 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-09-06T12:32:00.805 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.827 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.042) 0:06:47.220 ****** 2021-09-06T12:32:00.856 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:32:00.857 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.879 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.051) 0:06:47.272 ****** 2021-09-06T12:32:00.897 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-09-06T12:32:00.898 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.919 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.040) 0:06:47.312 ****** 2021-09-06T12:32:00.946 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:32:00.948 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:00.969 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:00 +0000 (0:00:00.049) 0:06:47.362 ****** 2021-09-06T12:32:00.986 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.008 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.039) 0:06:47.401 ****** 2021-09-06T12:32:01.034 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-06T12:32:01.036 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.057 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.048) 0:06:47.449 ****** 2021-09-06T12:32:01.074 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-09-06T12:32:01.076 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.097 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.039) 0:06:47.489 ****** 2021-09-06T12:32:01.123 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-06T12:32:01.125 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.146 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.049) 0:06:47.539 ****** 2021-09-06T12:32:01.163 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-09-06T12:32:01.165 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.186 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.039) 0:06:47.578 ****** 2021-09-06T12:32:01.212 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T12:32:01.214 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.237 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.051) 0:06:47.629 ****** 2021-09-06T12:32:01.254 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-09-06T12:32:01.256 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.278 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.040) 0:06:47.670 ****** 2021-09-06T12:32:01.304 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T12:32:01.306 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.326 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.048) 0:06:47.719 ****** 2021-09-06T12:32:01.342 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-09-06T12:32:01.344 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.365 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.038) 0:06:47.758 ****** 2021-09-06T12:32:01.392 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-06T12:32:01.394 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.415 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.050) 0:06:47.808 ****** 2021-09-06T12:32:01.433 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:01.434 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.456 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.040) 0:06:47.848 ****** 2021-09-06T12:32:01.482 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-09-06T12:32:01.484 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.507 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.051) 0:06:47.900 ****** 2021-09-06T12:32:01.524 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-09-06T12:32:01.525 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.546 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.038) 0:06:47.939 ****** 2021-09-06T12:32:01.573 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-06T12:32:01.574 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.596 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.049) 0:06:47.988 ****** 2021-09-06T12:32:01.614 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-09-06T12:32:01.616 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.637 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.040) 0:06:48.029 ****** 2021-09-06T12:32:01.664 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-06T12:32:01.666 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.687 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.049) 0:06:48.079 ****** 2021-09-06T12:32:01.705 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-09-06T12:32:01.706 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.728 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.041) 0:06:48.120 ****** 2021-09-06T12:32:01.765 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-06T12:32:01.767 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.788 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.060) 0:06:48.181 ****** 2021-09-06T12:32:01.795 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-09-06T12:32:01.796 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.818 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.029) 0:06:48.210 ****** 2021-09-06T12:32:01.855 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-06T12:32:01.856 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.877 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.058) 0:06:48.269 ****** 2021-09-06T12:32:01.883 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:32:01.885 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.906 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.029) 0:06:48.299 ****** 2021-09-06T12:32:01.943 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-06T12:32:01.945 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.966 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.059) 0:06:48.358 ****** 2021-09-06T12:32:01.972 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:32:01.974 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:01.996 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:01 +0000 (0:00:00.030) 0:06:48.389 ****** 2021-09-06T12:32:02.033 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-06T12:32:02.035 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.055 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.059) 0:06:48.448 ****** 2021-09-06T12:32:02.062 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:32:02.064 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.085 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.029) 0:06:48.478 ****** 2021-09-06T12:32:02.122 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-09-06T12:32:02.123 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.144 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.059) 0:06:48.537 ****** 2021-09-06T12:32:02.162 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T12:32:02.163 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.184 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.039) 0:06:48.577 ****** 2021-09-06T12:32:02.212 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:02.213 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-06T12:32:02.235 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.050) 0:06:48.627 ****** 2021-09-06T12:32:02.251 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T12:32:02.253 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.275 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.039) 0:06:48.667 ****** 2021-09-06T12:32:02.312 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-06T12:32:02.314 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.323 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-06T12:32:02.334 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.336 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.346 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.347 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi041.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.371 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.096) 0:06:48.764 ****** 2021-09-06T12:32:02.378 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.405 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.033) 0:06:48.797 ****** 2021-09-06T12:32:02.432 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:32:02.434 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.455 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.049) 0:06:48.847 ****** 2021-09-06T12:32:02.513 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:32:02.515 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.535 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.079) 0:06:48.927 ****** 2021-09-06T12:32:02.602 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:32:02.604 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.624 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.089) 0:06:49.017 ****** 2021-09-06T12:32:02.631 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-06T12:32:02.633 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003164", "end": "2021-09-06 12:32:02.603977", "rc": 0, "start": "2021-09-06 12:32:02.600813", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-06T12:32:02.658 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.033) 0:06:49.050 ****** 2021-09-06T12:32:02.685 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-09-06T12:32:02.687 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.708 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.049) 0:06:49.100 ****** 2021-09-06T12:32:02.724 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-06T12:32:02.726 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.750 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.042) 0:06:49.143 ****** 2021-09-06T12:32:02.767 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-09-06T12:32:02.769 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.791 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.040) 0:06:49.183 ****** 2021-09-06T12:32:02.818 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-06T12:32:02.819 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.841 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.049) 0:06:49.233 ****** 2021-09-06T12:32:02.857 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-09-06T12:32:02.859 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.880 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.039) 0:06:49.272 ****** 2021-09-06T12:32:02.907 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-06T12:32:02.909 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.930 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.049) 0:06:49.322 ****** 2021-09-06T12:32:02.946 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-09-06T12:32:02.948 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:02.969 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:02 +0000 (0:00:00.039) 0:06:49.362 ****** 2021-09-06T12:32:02.995 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-06T12:32:02.997 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.018 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.048) 0:06:49.410 ****** 2021-09-06T12:32:03.035 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-09-06T12:32:03.037 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.058 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.039) 0:06:49.450 ****** 2021-09-06T12:32:03.125 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-09-06T12:32:03.126 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.146 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.088) 0:06:49.538 ****** 2021-09-06T12:32:03.213 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-09-06T12:32:03.214 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.234 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.087) 0:06:49.626 ****** 2021-09-06T12:32:03.300 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-09-06T12:32:03.302 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.322 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.087) 0:06:49.714 ****** 2021-09-06T12:32:03.388 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.412 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.090) 0:06:49.805 ****** 2021-09-06T12:32:03.468 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-09-06T12:32:03.470 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.489 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.077) 0:06:49.882 ****** 2021-09-06T12:32:03.546 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-09-06T12:32:03.547 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.567 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.077) 0:06:49.959 ****** 2021-09-06T12:32:03.633 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-09-06T12:32:03.635 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.655 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.087) 0:06:50.047 ****** 2021-09-06T12:32:03.712 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-09-06T12:32:03.714 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.738 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.082) 0:06:50.130 ****** 2021-09-06T12:32:03.806 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-09-06T12:32:03.807 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.827 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.089) 0:06:50.220 ****** 2021-09-06T12:32:03.895 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-09-06T12:32:03.897 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:03.916 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:03 +0000 (0:00:00.088) 0:06:50.309 ****** 2021-09-06T12:32:03.984 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.005 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.088) 0:06:50.397 ****** 2021-09-06T12:32:04.073 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:04.074 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-09-06T12:32:04.094 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.088) 0:06:50.486 ****** 2021-09-06T12:32:04.172 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-09-06T12:32:04.174 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} ok: [smithi052.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-09-06T12:32:04.184 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.184 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.195 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.196 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.206 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.231 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.136) 0:06:50.623 ****** 2021-09-06T12:32:04.444 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-09-06T12:32:04.445 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002941", "end": "2021-09-06 12:32:04.423472", "rc": 0, "start": "2021-09-06 12:32:04.420531", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology-lock"]} 2021-09-06T12:32:04.467 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.236) 0:06:50.860 ****** 2021-09-06T12:32:04.536 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-09-06T12:32:04.537 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.559 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.091) 0:06:50.952 ****** 2021-09-06T12:32:04.636 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-09-06T12:32:04.638 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.661 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.102) 0:06:51.054 ****** 2021-09-06T12:32:04.729 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-09-06T12:32:04.731 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.751 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.089) 0:06:51.143 ****** 2021-09-06T12:32:04.807 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-09-06T12:32:04.809 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:04.830 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:04 +0000 (0:00:00.079) 0:06:51.222 ****** 2021-09-06T12:32:06.158 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-06T12:32:06.160 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:32:06.181 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:06 +0000 (0:00:01.351) 0:06:52.574 ****** 2021-09-06T12:32:06.261 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T12:32:06.263 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-06T12:32:06.284 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:06 +0000 (0:00:00.102) 0:06:52.676 ****** 2021-09-06T12:32:06.518 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-06T12:32:06.519 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005448", "end": "2021-09-06 12:32:06.495192", "rc": 0, "start": "2021-09-06 12:32:06.489744", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:06.539 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:06 +0000 (0:00:00.255) 0:06:52.932 ****** 2021-09-06T12:32:06.617 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-06T12:32:06.618 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:06.640 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:06 +0000 (0:00:00.100) 0:06:53.032 ****** 2021-09-06T12:32:06.708 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-06T12:32:06.728 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:06 +0000 (0:00:00.088) 0:06:53.121 ****** 2021-09-06T12:32:07.977 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-09-06T12:32:07.979 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:32:07.999 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:07 +0000 (0:00:01.270) 0:06:54.391 ****** 2021-09-06T12:32:08.077 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T12:32:08.078 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}} 2021-09-06T12:32:08.099 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:08 +0000 (0:00:00.100) 0:06:54.492 ****** 2021-09-06T12:32:08.125 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-06T12:32:08.126 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLj1HthQXQb2iVT6IU4heUnr22PRzog84ph'], 'uuids': ['3d6652a1-0758-42f6-9dd3-21a76db97123'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.142106", "end": "2021-09-06 12:32:08.095687", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLj1HthQXQb2iVT6IU4heUnr22PRzog84ph"], "labels": [], "masters": [], "uuids": ["3d6652a1-0758-42f6-9dd3-21a76db97123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:06.953581", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:08.334 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-09-06T12:32:08.336 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005475", "end": "2021-09-06 12:32:08.310181", "rc": 0, "start": "2021-09-06 12:32:08.304706", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:08.356 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:08 +0000 (0:00:00.257) 0:06:54.749 ****** 2021-09-06T12:32:08.435 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-09-06T12:32:08.436 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:08.456 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:08 +0000 (0:00:00.099) 0:06:54.849 ****** 2021-09-06T12:32:08.513 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-09-06T12:32:08.533 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:08 +0000 (0:00:00.076) 0:06:54.926 ****** 2021-09-06T12:32:09.384 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-epAysS-Mdc0-shWs-LMAi-cZFA-gwvY-xIDpnx', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN', 'nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.068319", "end": "2021-09-06 12:32:09.359715", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-epAysS-Mdc0-shWs-LMAi-cZFA-gwvY-xIDpnx", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN", "nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:08.291396", "stderr": "", "stde 2021-09-06T12:32:09.386 INFO:teuthology.task.ansible.out:rr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:09.934 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2K7I4q4syxcwlikud2qCpddGeZQ4jpJeq'], 'uuids': ['4c25f5d7-6f14-4be4-9da1-97d573b33b8e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.151282", "end": "2021-09-06 12:32:09.910561", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2K7I4q4syxcwlikud2qCpddGeZQ4jpJeq"], "labels": [], "masters": [], "uuids": ["4c25f5d7-6f14-4be4-9da1-97d573b33b8e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:08.759279", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:10.630 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjcX6nis23GQI7S8oD1QcG2ty2t15hf5oI'], 'uuids': ['4234acc6-a676-41bb-a6d2-54b217447997'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.054861", "end": "2021-09-06 12:32:10.603296", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjcX6nis23GQI7S8oD1QcG2ty2t15hf5oI"], "labels": [], "masters": [], "uuids": ["4234acc6-a676-41bb-a6d2-54b217447997"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:09.548435", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:11.179 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-KuR2fE-rTtG-nDgc-lGy0-Hx1R-8WqZ-gDSYks', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.048686", "end": "2021-09-06 12:32:11.148190", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-KuR2fE-rTtG-nDgc-lGy0-Hx1R-8WqZ-gDSYks", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:10.099504", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-09-06T12:32:11.181 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:11.894 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjGHqEZKxZg9cFCHR7O81idxReQfeJi8tY'], 'uuids': ['223d1cc0-dec4-44fb-9c77-5bec37d66c46'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.067068", "end": "2021-09-06 12:32:11.862625", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjGHqEZKxZg9cFCHR7O81idxReQfeJi8tY"], "labels": [], "masters": [], "uuids": ["223d1cc0-dec4-44fb-9c77-5bec37d66c46"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:10.795557", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:12.421 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2KC5AidJNYd6fBnw0fkKSpUyFNlYfTuLv'], 'uuids': ['c6e7e62c-06f7-4aa0-842b-f4086c20a935'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.046401", "end": "2021-09-06 12:32:12.386387", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2KC5AidJNYd6fBnw0fkKSpUyFNlYfTuLv"], "labels": [], "masters": [], "uuids": ["c6e7e62c-06f7-4aa0-842b-f4086c20a935"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:11.339986", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:13.145 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjLw9uEQwNFN1GWUWoxujX2KrmIibtQezN'], 'uuids': ['2856bc0b-d767-44f7-81c8-80fb515fe292'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.062751", "end": "2021-09-06 12:32:13.113271", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjLw9uEQwNFN1GWUWoxujX2KrmIibtQezN"], "labels": [], "masters": [], "uuids": ["2856bc0b-d767-44f7-81c8-80fb515fe292"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:12.050520", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:13.147 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S', 'scsi-35000c500871a27d9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S', 'wwn-0x5000c500871a27d9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HM0S', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a27d9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'scsi-35000c500871a27d9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S-part1', 'wwn-0x5000c500871a27d9-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S", "scsi-35000c500871a27d9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S", "wwn-0x5000c500871a27d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T12:32:13.148 INFO:teuthology.task.ansible.out:Z1W4HM0S-part1", "scsi-35000c500871a27d9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S-part1", "wwn-0x5000c500871a27d9-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HM0S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a27d9"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:13.685 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2cBfh9AXRTpuXHsv4kyrC8WlqQtZRS0JY'], 'uuids': ['05e84556-de2d-4004-9154-e3f2be96e9bc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.077555", "end": "2021-09-06 12:32:13.658803", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2cBfh9AXRTpuXHsv4kyrC8WlqQtZRS0JY"], "labels": [], "masters": [], "uuids": ["05e84556-de2d-4004-9154-e3f2be96e9bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:12.581248", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:14.379 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjvwJpn3CS1049cVtBk5YAcQ3ViMglTe2i'], 'uuids': ['4e75702d-be28-47ff-87af-79601d505a80'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.042847", "end": "2021-09-06 12:32:14.355074", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjvwJpn3CS1049cVtBk5YAcQ3ViMglTe2i"], "labels": [], "masters": [], "uuids": ["4e75702d-be28-47ff-87af-79601d505a80"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:13.312227", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:14.416 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:14 +0000 (0:00:05.882) 0:07:00.809 ****** 2021-09-06T12:32:14.825 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:14.827 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLj1HthQXQb2iVT6IU4heUnr22PRzog84ph'], 'uuids': ['3d6652a1-0758-42f6-9dd3-21a76db97123'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.145077", "end": "2021-09-06 12:32:14.798571", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLj1HthQXQb2iVT6IU4heUnr22PRzog84ph"], "labels": [], "masters": [], "uuids": ["3d6652a1-0758-42f6-9dd3-21a76db97123"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:14.653494", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142137 s, 811 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142137 s, 811 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:14.920 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-06T12:32:14.921 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2oWn1u5Q1d7VDlNtdYZc5kwNRZF1Pof1q'], 'uuids': ['5c6cf814-b4cb-4828-9795-3f214c414dbf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.044573", "end": "2021-09-06 12:32:14.894888", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2oWn1u5Q1d7VDlNtdYZc5kwNRZF1Pof1q"], "labels": [], "masters": [], "uuids": ["5c6cf814-b4cb-4828-9795-3f214c414dbf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:13.850315", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:14.932 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS', 'scsi-35000c500871a48d9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HRYS', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a48d9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'scsi-35000c500871a48d9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS", "scsi-35000c500871a48d9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T12:32:14.933 INFO:teuthology.task.ansible.out:Z1W4HRYS-part1", "scsi-35000c500871a48d9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:15.036 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:15.038 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-epAysS-Mdc0-shWs-LMAi-cZFA-gwvY-xIDpnx', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN', 'nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039496", "end": "2021-09-06 12:32:15.016531", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-epAysS-Mdc0-shWs-LMAi-cZFA-gwvY-xIDpnx", "nvme-INTEL_SSDPEDMD400G4_CVFT5331003M400BGN", "nvme-nvme.8086-43564654353333313030334d34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:14.977035", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0366952 s, 3.1 GB/s", "std 2021-09-06T12:32:15.040 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0366952 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:15.360 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjcX6nis23GQI7S8oD1QcG2ty2t15hf5oI'], 'uuids': ['4234acc6-a676-41bb-a6d2-54b217447997'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.144921", "end": "2021-09-06 12:32:15.336231", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjcX6nis23GQI7S8oD1QcG2ty2t15hf5oI"], "labels": [], "masters": [], "uuids": ["4234acc6-a676-41bb-a6d2-54b217447997"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:15.191310", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142056 s, 812 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142056 s, 812 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:15.681 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjGHqEZKxZg9cFCHR7O81idxReQfeJi8tY'], 'uuids': ['223d1cc0-dec4-44fb-9c77-5bec37d66c46'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142867", "end": "2021-09-06 12:32:15.655594", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjGHqEZKxZg9cFCHR7O81idxReQfeJi8tY"], "labels": [], "masters": [], "uuids": ["223d1cc0-dec4-44fb-9c77-5bec37d66c46"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:15.512727", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139991 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139991 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:15.993 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjLw9uEQwNFN1GWUWoxujX2KrmIibtQezN'], 'uuids': ['2856bc0b-d767-44f7-81c8-80fb515fe292'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144177", "end": "2021-09-06 12:32:15.973345", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjLw9uEQwNFN1GWUWoxujX2KrmIibtQezN"], "labels": [], "masters": [], "uuids": ["2856bc0b-d767-44f7-81c8-80fb515fe292"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:15.829168", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141417 s, 816 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141417 s, 816 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.004 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S', 'scsi-35000c500871a27d9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S', 'wwn-0x5000c500871a27d9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HM0S', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a27d9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S-part1', 'scsi-35000c500871a27d9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S-part1', 'wwn-0x5000c500871a27d9-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HM0S", "scsi-35000c500871a27d9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S", "wwn-0x5000c500871a27d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HM0S-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HM0S-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T12:32:16.006 INFO:teuthology.task.ansible.out:Z1W4HM0S-part1", "scsi-35000c500871a27d9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HM0S-part1", "wwn-0x5000c500871a27d9-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HM0S", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a27d9"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:16.160 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-09-06T12:32:16.162 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2jp4a6VjRDXibtc30f2NpQ1ndT0wXqCNf'], 'uuids': ['ef1f8bea-d149-4b6b-a20f-b53293b27db6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.043137", "end": "2021-09-06 12:32:16.135749", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2jp4a6VjRDXibtc30f2NpQ1ndT0wXqCNf"], "labels": [], "masters": [], "uuids": ["ef1f8bea-d149-4b6b-a20f-b53293b27db6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:15.092612", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-09-06T12:32:16.200 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:16 +0000 (0:00:01.783) 0:07:02.592 ****** 2021-09-06T12:32:16.319 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:16.321 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjvwJpn3CS1049cVtBk5YAcQ3ViMglTe2i'], 'uuids': ['4e75702d-be28-47ff-87af-79601d505a80'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.142246", "end": "2021-09-06 12:32:16.297621", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-UI9g5630gGXPlpqpcDjMmkDXKNRSVWLjvwJpn3CS1049cVtBk5YAcQ3ViMglTe2i"], "labels": [], "masters": [], "uuids": ["4e75702d-be28-47ff-87af-79601d505a80"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:16.155375", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139407 s, 827 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139407 s, 827 MB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.347 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:16 +0000 (0:00:00.147) 0:07:02.740 ****** 2021-09-06T12:32:16.563 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2K7I4q4syxcwlikud2qCpddGeZQ4jpJeq'], 'uuids': ['4c25f5d7-6f14-4be4-9da1-97d573b33b8e'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.100340", "end": "2021-09-06 12:32:16.532901", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2K7I4q4syxcwlikud2qCpddGeZQ4jpJeq"], "labels": [], "masters": [], "uuids": ["4c25f5d7-6f14-4be4-9da1-97d573b33b8e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:16.432561", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0974587 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0974587 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.628 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-06T12:32:16.629 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.054788", "end": "2021-09-06 12:32:16.565227", "rc": 0, "start": "2021-09-06 12:32:16.510439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.659 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:16 +0000 (0:00:00.311) 0:07:03.051 ****** 2021-09-06T12:32:16.772 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:16.773 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-KuR2fE-rTtG-nDgc-lGy0-Hx1R-8WqZ-gDSYks', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPED1D480GA', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '937703088', 'sectorsize': '512', 'size': '447.13 GB', 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.039246", "end": "2021-09-06 12:32:16.748759", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-KuR2fE-rTtG-nDgc-lGy0-Hx1R-8WqZ-gDSYks", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:16.709513", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0363936 s, 3.2 GB/s", "stderr_lines": [" 2021-09-06T12:32:16.775 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0363936 s, 3.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.897 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.050776", "end": "2021-09-06 12:32:16.876161", "rc": 0, "start": "2021-09-06 12:32:16.825385", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:16.918 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:16 +0000 (0:00:00.259) 0:07:03.311 ****** 2021-09-06T12:32:16.976 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-06T12:32:17.029 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.111) 0:07:03.422 ****** 2021-09-06T12:32:17.046 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:17.046 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2KC5AidJNYd6fBnw0fkKSpUyFNlYfTuLv'], 'uuids': ['c6e7e62c-06f7-4aa0-842b-f4086c20a935'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.100712", "end": "2021-09-06 12:32:17.026054", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2KC5AidJNYd6fBnw0fkKSpUyFNlYfTuLv"], "labels": [], "masters": [], "uuids": ["c6e7e62c-06f7-4aa0-842b-f4086c20a935"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:16.925342", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0979386 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0979386 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:17.057 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-06T12:32:17.059 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.081 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.051) 0:07:03.473 ****** 2021-09-06T12:32:17.106 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-06T12:32:17.108 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.127 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.046) 0:07:03.519 ****** 2021-09-06T12:32:17.152 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T12:32:17.154 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.175 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.047) 0:07:03.567 ****** 2021-09-06T12:32:17.201 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-06T12:32:17.202 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.223 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.048) 0:07:03.616 ****** 2021-09-06T12:32:17.322 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:17.324 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2cBfh9AXRTpuXHsv4kyrC8WlqQtZRS0JY'], 'uuids': ['05e84556-de2d-4004-9154-e3f2be96e9bc'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.100492", "end": "2021-09-06 12:32:17.300064", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2cBfh9AXRTpuXHsv4kyrC8WlqQtZRS0JY"], "labels": [], "masters": [], "uuids": ["05e84556-de2d-4004-9154-e3f2be96e9bc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:17.199572", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0977326 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0977326 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:17.417 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-06T12:32:17.418 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:32:17.438 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.214) 0:07:03.830 ****** 2021-09-06T12:32:17.474 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T12:32:17.475 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.496 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.058) 0:07:03.888 ****** 2021-09-06T12:32:17.522 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.544 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.047) 0:07:03.936 ****** 2021-09-06T12:32:17.570 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:17.572 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.593 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.049) 0:07:03.985 ****** 2021-09-06T12:32:17.601 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:17.601 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2oWn1u5Q1d7VDlNtdYZc5kwNRZF1Pof1q'], 'uuids': ['5c6cf814-b4cb-4828-9795-3f214c414dbf'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.100273", "end": "2021-09-06 12:32:17.574257", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2oWn1u5Q1d7VDlNtdYZc5kwNRZF1Pof1q"], "labels": [], "masters": [], "uuids": ["5c6cf814-b4cb-4828-9795-3f214c414dbf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:17.473984", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0975744 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0975744 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:17.613 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS', 'scsi-35000c500871a48d9', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W4HRYS', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500871a48d9', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'scsi-35000c500871a48d9-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['bc802139-74bd-43cf-840d-e30da2bb57f8'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HRYS", "scsi-35000c500871a48d9", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W4HRYS-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-09-06T12:32:17.615 INFO:teuthology.task.ansible.out:Z1W4HRYS-part1", "scsi-35000c500871a48d9-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["bc802139-74bd-43cf-840d-e30da2bb57f8"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.625 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:17.626 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.647 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.054) 0:07:04.039 ****** 2021-09-06T12:32:17.673 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-06T12:32:17.675 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.695 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.048) 0:07:04.088 ****** 2021-09-06T12:32:17.721 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-09-06T12:32:17.723 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.743 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.047) 0:07:04.136 ****** 2021-09-06T12:32:17.770 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-06T12:32:17.772 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:17.791 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.048) 0:07:04.184 ****** 2021-09-06T12:32:17.882 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-09-06T12:32:17.884 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2jp4a6VjRDXibtc30f2NpQ1ndT0wXqCNf'], 'uuids': ['ef1f8bea-d149-4b6b-a20f-b53293b27db6'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.100131", "end": "2021-09-06 12:32:17.858071", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-S9WVQSiYSdj8qasKllpaLfjRH4jCDcg2jp4a6VjRDXibtc30f2NpQ1ndT0wXqCNf"], "labels": [], "masters": [], "uuids": ["ef1f8bea-d149-4b6b-a20f-b53293b27db6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-09-06 12:32:17.757940", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0973582 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0973582 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:17.911 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:17 +0000 (0:00:00.119) 0:07:04.303 ****** 2021-09-06T12:32:18.179 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-09-06T12:32:18.181 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.080301", "end": "2021-09-06 12:32:18.156888", "rc": 0, "start": "2021-09-06 12:32:18.076587", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:18.200 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.288) 0:07:04.592 ****** 2021-09-06T12:32:18.309 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-09-06T12:32:18.311 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-06T12:32:18.333 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.132) 0:07:04.725 ****** 2021-09-06T12:32:18.422 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-09-06T12:32:18.423 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.041729", "end": "2021-09-06 12:32:18.403982", "rc": 0, "start": "2021-09-06 12:32:18.362253", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:18.447 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.114) 0:07:04.839 ****** 2021-09-06T12:32:18.474 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-09-06T12:32:18.497 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.050) 0:07:04.889 ****** 2021-09-06T12:32:18.523 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-09-06T12:32:18.525 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:18.546 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.049) 0:07:04.939 ****** 2021-09-06T12:32:18.572 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-09-06T12:32:18.574 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:18.593 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.046) 0:07:04.986 ****** 2021-09-06T12:32:18.619 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T12:32:18.621 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:18.643 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.049) 0:07:05.035 ****** 2021-09-06T12:32:18.669 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-09-06T12:32:18.671 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:18.692 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.049) 0:07:05.085 ****** 2021-09-06T12:32:18.782 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:18.784 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:18.887 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-09-06T12:32:18.889 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-09-06T12:32:18.909 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.216) 0:07:05.301 ****** 2021-09-06T12:32:18.945 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-09-06T12:32:18.946 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930320781312, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227129097, 'block_used': 12966011, 'inode_total': 61054976, 'inode_available': 60985432, 'inode_used': 69544, 'uuid': 'bc802139-74bd-43cf-840d-e30da2bb57f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227129097, "block_size": 4096, "block_total": 240095108, "block_used": 12966011, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985432, "inode_total": 61054976, "inode_used": 69544, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930320781312, "size_total": 983429562368, "uuid": "bc802139-74bd-43cf-840d-e30da2bb57f8"}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:18.967 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:18 +0000 (0:00:00.057) 0:07:05.359 ****** 2021-09-06T12:32:18.993 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-09-06T12:32:18.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.013 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.046) 0:07:05.406 ****** 2021-09-06T12:32:19.039 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:19.041 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.061 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.047) 0:07:05.453 ****** 2021-09-06T12:32:19.086 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:19.088 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.107 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.046) 0:07:05.500 ****** 2021-09-06T12:32:19.123 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:19.125 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:19.135 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-09-06T12:32:19.136 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.156 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.048) 0:07:05.548 ****** 2021-09-06T12:32:19.181 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.201 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.045) 0:07:05.594 ****** 2021-09-06T12:32:19.227 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-09-06T12:32:19.229 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:19.247 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.045) 0:07:05.640 ****** 2021-09-06T12:32:19.491 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:19.493 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:19.652 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-09-06T12:32:19.654 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-09-06T12:32:19.674 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:19 +0000 (0:00:00.426) 0:07:06.066 ****** 2021-09-06T12:32:19.846 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:19.848 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:20.055 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:20.263 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:20.287 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:20 +0000 (0:00:00.612) 0:07:06.679 ****** 2021-09-06T12:32:20.386 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:20.481 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T12:32:20.483 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-06T12:32:20.503 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:20 +0000 (0:00:00.216) 0:07:06.895 ****** 2021-09-06T12:32:20.717 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:20.718 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:20.759 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:20.761 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:20.896 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:20.897 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:21.064 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:21.106 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:21.108 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:21.253 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:21.255 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi041.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:21.277 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.774) 0:07:07.669 ****** 2021-09-06T12:32:21.303 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-06T12:32:21.305 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:21.324 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.046) 0:07:07.716 ****** 2021-09-06T12:32:21.350 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-06T12:32:21.352 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:21.371 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.047) 0:07:07.764 ****** 2021-09-06T12:32:21.439 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-09-06T12:32:21.440 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-09-06T12:32:21.463 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.091) 0:07:07.856 ****** 2021-09-06T12:32:21.573 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-09-06T12:32:21.575 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:21.596 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.132) 0:07:07.988 ****** 2021-09-06T12:32:21.623 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-06T12:32:21.624 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:21.645 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.049) 0:07:08.037 ****** 2021-09-06T12:32:21.661 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-09-06T12:32:21.662 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-09-06T12:32:21.684 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:21 +0000 (0:00:00.039) 0:07:08.077 ****** 2021-09-06T12:32:21.908 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:21.910 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:22.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-06T12:32:22.015 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true} 2021-09-06T12:32:22.038 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.353) 0:07:08.431 ****** 2021-09-06T12:32:22.077 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:22.078 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:22.265 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-09-06T12:32:22.421 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-06T12:32:22.443 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.404) 0:07:08.835 ****** 2021-09-06T12:32:22.451 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-09-06T12:32:22.452 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.476 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.033) 0:07:08.869 ****** 2021-09-06T12:32:22.482 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-06T12:32:22.483 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.504 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:08.897 ****** 2021-09-06T12:32:22.510 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-09-06T12:32:22.512 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.532 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:08.925 ****** 2021-09-06T12:32:22.538 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-06T12:32:22.540 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.560 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:08.953 ****** 2021-09-06T12:32:22.566 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-09-06T12:32:22.568 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.589 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:08.981 ****** 2021-09-06T12:32:22.594 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T12:32:22.596 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.617 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:09.010 ****** 2021-09-06T12:32:22.643 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T12:32:22.645 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.665 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.047) 0:07:09.057 ****** 2021-09-06T12:32:22.691 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.713 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.048) 0:07:09.105 ****** 2021-09-06T12:32:22.740 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-06T12:32:22.741 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.762 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.048) 0:07:09.154 ****** 2021-09-06T12:32:22.788 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-06T12:32:22.790 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:22.811 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.048) 0:07:09.203 ****** 2021-09-06T12:32:22.829 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.018) 0:07:09.221 ****** 2021-09-06T12:32:22.836 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-09-06T12:32:22.838 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.859 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.030) 0:07:09.252 ****** 2021-09-06T12:32:22.866 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-09-06T12:32:22.868 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.888 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.028) 0:07:09.281 ****** 2021-09-06T12:32:22.894 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-09-06T12:32:22.896 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.921 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.032) 0:07:09.313 ****** 2021-09-06T12:32:22.948 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-09-06T12:32:22.949 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:22.969 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:22 +0000 (0:00:00.048) 0:07:09.361 ****** 2021-09-06T12:32:22.994 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-09-06T12:32:22.996 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.016 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:09.409 ****** 2021-09-06T12:32:23.043 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-06T12:32:23.044 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.063 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.046) 0:07:09.456 ****** 2021-09-06T12:32:23.089 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-06T12:32:23.091 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.111 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:09.503 ****** 2021-09-06T12:32:23.127 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-09-06T12:32:23.129 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true} 2021-09-06T12:32:23.149 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.038) 0:07:09.541 ****** 2021-09-06T12:32:23.155 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-06T12:32:23.157 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.178 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.028) 0:07:09.570 ****** 2021-09-06T12:32:23.204 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-06T12:32:23.206 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.226 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:09.618 ****** 2021-09-06T12:32:23.449 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-09-06T12:32:23.451 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-09-06T12:32:23.472 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.246) 0:07:09.865 ****** 2021-09-06T12:32:23.499 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-09-06T12:32:23.501 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.520 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.048) 0:07:09.913 ****** 2021-09-06T12:32:23.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-09-06T12:32:23.548 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.568 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:09.960 ****** 2021-09-06T12:32:23.593 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T12:32:23.595 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.614 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.046) 0:07:10.007 ****** 2021-09-06T12:32:23.639 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-09-06T12:32:23.641 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.661 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.046) 0:07:10.053 ****** 2021-09-06T12:32:23.686 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-09-06T12:32:23.688 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.708 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:10.101 ****** 2021-09-06T12:32:23.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-06T12:32:23.717 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-06T12:32:23.738 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.029) 0:07:10.130 ****** 2021-09-06T12:32:23.744 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-09-06T12:32:23.746 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.768 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.030) 0:07:10.161 ****** 2021-09-06T12:32:23.795 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-09-06T12:32:23.797 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.817 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.048) 0:07:10.209 ****** 2021-09-06T12:32:23.854 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-09-06T12:32:23.855 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.881 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.063) 0:07:10.273 ****** 2021-09-06T12:32:23.912 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-09-06T12:32:23.914 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.934 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.052) 0:07:10.326 ****** 2021-09-06T12:32:23.959 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:23.981 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:23 +0000 (0:00:00.047) 0:07:10.374 ****** 2021-09-06T12:32:24.009 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.031 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.050) 0:07:10.424 ****** 2021-09-06T12:32:24.049 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-06T12:32:24.051 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-09-06 12:24:14 UTC", "ActiveEnterTimestampMonotonic": "18016259", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target tmp.mount basic.target -.mount system.slice ntpdate.service ntpd.service sntp.service systemd-journald.socket systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-09-06 12:24:13 UTC", "AssertTimestampMonotonic": "16754383", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-09-06 12:24:13 UTC", "ConditionTimestampMonotonic": "16754317", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-time 2021-09-06T12:32:24.052 INFO:teuthology.task.ansible.out:syncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "868", "ExecMainStartTimestamp": "Mon 2021-09-06 12:24:14 UTC", "ExecMainStartTimestampMonotonic": "17924930", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-09-06 12:24:13 UTC", "InactiveExitTimestampMonotonic": "16763200", "InvocationID": "e7f373132489470290f8ac3e66a21db7", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-06T12:32:24.054 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "868", "MemoryAccounting": "yes", "MemoryCurrent": "1253376", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit. 2021-09-06T12:32:24.056 INFO:teuthology.task.ansible.out:target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-09-06 12:24:14 UTC", "StateChangeTimestampMonotonic": "18016259", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-09-06 12:24:14 UTC", "WatchdogTimestampMonotonic": "17924971", "WatchdogUSec": "0"}} 2021-09-06T12:32:24.073 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.042) 0:07:10.466 ****** 2021-09-06T12:32:24.080 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-09-06T12:32:24.081 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.102 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.029) 0:07:10.495 ****** 2021-09-06T12:32:24.109 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:24.111 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-06T12:32:24.132 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.029) 0:07:10.524 ****** 2021-09-06T12:32:24.138 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-09-06T12:32:24.140 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.161 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.028) 0:07:10.553 ****** 2021-09-06T12:32:24.167 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:24.169 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-06T12:32:24.191 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.029) 0:07:10.583 ****** 2021-09-06T12:32:24.196 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-09-06T12:32:24.198 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.218 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.027) 0:07:10.610 ****** 2021-09-06T12:32:24.223 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-06T12:32:24.225 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.246 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.028) 0:07:10.638 ****** 2021-09-06T12:32:24.252 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-09-06T12:32:24.253 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.275 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.029) 0:07:10.667 ****** 2021-09-06T12:32:24.343 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 06 September 2021 12:32:24 +0000 (0:00:00.030) 0:07:10.698 ****** TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:24.354 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.048) 0:07:10.746 ****** 2021-09-06T12:32:24.557 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-06T12:32:24.559 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-06T12:32:24.579 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.225) 0:07:10.971 ****** 2021-09-06T12:32:24.750 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-09-06T12:32:24.752 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-09-06T12:32:24.774 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:24 +0000 (0:00:00.194) 0:07:11.166 ****** 2021-09-06T12:32:25.048 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-09-06T12:32:25.050 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-06T12:32:25.076 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.302) 0:07:11.469 ****** 2021-09-06T12:32:25.086 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-09-06T12:32:25.087 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-09-06 12:24:07 UTC", "ActiveEnterTimestampMonotonic": "18227405", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-tmpfiles-setup.service system.slice systemd-journald.socket tmp.mount sysinit.target sntp.service ntpdate.service ntpd.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-09-06 12:24:06 UTC", "AssertTimestampMonotonic": "16753913", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUAffinityFromNUMA": "no", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanFreeze": "yes", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-09-06 12:24:06 UTC", "ConditionTimestampMonotonic": "16753831", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.tar 2021-09-06T12:32:25.089 INFO:teuthology.task.ansible.out:get systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "851", "ExecMainStartTimestamp": "Mon 2021-09-06 12:24:06 UTC", "ExecMainStartTimestampMonotonic": "16891967", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "FreezerState": "running", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-09-06 12:24:06 UTC", "InactiveExitTimestampMonotonic": "16757993", "InvocationID": "1a493e7a48c5458290f5f65247ba5d67", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinit 2021-09-06T12:32:25.091 INFO:teuthology.task.ansible.out:y", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127264", "LimitNPROCSoft": "127264", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127264", "LimitSIGPENDINGSoft": "127264", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "851", "MemoryAccounting": "yes", "MemoryCurrent": "1261568", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice - 2021-09-06T12:32:25.092 INFO:teuthology.task.ansible.out:.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-09-06 12:24:07 UTC", "StateChangeTimestampMonotonic": "18227405", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203622", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-09-06 12:24:06 UTC", "WatchdogTimestampMonotonic": "16892012", "WatchdogUSec": "0"}} 2021-09-06T12:32:25.112 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.035) 0:07:11.504 ****** 2021-09-06T12:32:25.138 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:25.140 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-09-06T12:32:25.160 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.048) 0:07:11.552 ****** 2021-09-06T12:32:25.196 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:25.198 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-09-06T12:32:25.217 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.057) 0:07:11.610 ****** 2021-09-06T12:32:25.243 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-09-06T12:32:25.245 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:25.265 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.047) 0:07:11.658 ****** 2021-09-06T12:32:25.292 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-09-06T12:32:25.293 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:25.313 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.047) 0:07:11.705 ****** 2021-09-06T12:32:25.339 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-09-06T12:32:25.341 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:25.361 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.047) 0:07:11.753 ****** 2021-09-06T12:32:25.564 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-09-06T12:32:25.566 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-09-06T12:32:25.586 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:25 +0000 (0:00:00.225) 0:07:11.978 ****** 2021-09-06T12:32:26.050 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-09-06T12:32:26.052 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-09-06T12:32:26.072 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.486) 0:07:12.465 ****** 2021-09-06T12:32:26.255 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-06T12:32:26.256 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-06T12:32:26.278 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.205) 0:07:12.670 ****** 2021-09-06T12:32:26.304 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-06T12:32:26.305 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:26.325 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.047) 0:07:12.717 ****** 2021-09-06T12:32:26.351 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-06T12:32:26.353 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-06T12:32:26.372 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.047) 0:07:12.764 ****** 2021-09-06T12:32:26.554 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-06T12:32:26.556 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-06T12:32:26.576 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.203) 0:07:12.968 ****** 2021-09-06T12:32:26.603 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-06T12:32:26.604 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:26.628 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.052) 0:07:13.021 ****** 2021-09-06T12:32:26.655 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-06T12:32:26.657 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-06T12:32:26.676 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.047) 0:07:13.068 ****** 2021-09-06T12:32:26.703 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-06T12:32:26.704 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:26.724 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.048) 0:07:13.117 ****** 2021-09-06T12:32:26.917 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-06T12:32:26.918 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-06T12:32:26.938 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:26 +0000 (0:00:00.213) 0:07:13.331 ****** 2021-09-06T12:32:27.131 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-06T12:32:27.133 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-06T12:32:27.152 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.214) 0:07:13.545 ****** 2021-09-06T12:32:27.159 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-09-06T12:32:27.161 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-09-06T12:32:27.182 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.029) 0:07:13.574 ****** 2021-09-06T12:32:27.187 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-06T12:32:27.189 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.210 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.027) 0:07:13.602 ****** 2021-09-06T12:32:27.216 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-09-06T12:32:27.218 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.240 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.029) 0:07:13.632 ****** 2021-09-06T12:32:27.256 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-06T12:32:27.258 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.259 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.280 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.040) 0:07:13.672 ****** 2021-09-06T12:32:27.286 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-09-06T12:32:27.288 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-09-06T12:32:27.310 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.029) 0:07:13.702 ****** 2021-09-06T12:32:27.317 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-06T12:32:27.319 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.330 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.332 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi041.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.356 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.046) 0:07:13.749 ****** 2021-09-06T12:32:27.383 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-09-06T12:32:27.385 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.407 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.050) 0:07:13.800 ****** 2021-09-06T12:32:27.433 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-06T12:32:27.435 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.454 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.047) 0:07:13.847 ****** 2021-09-06T12:32:27.480 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-06T12:32:27.482 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.504 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.049) 0:07:13.896 ****** 2021-09-06T12:32:27.510 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-09-06T12:32:27.511 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-09-06T12:32:27.532 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.028) 0:07:13.925 ****** 2021-09-06T12:32:27.537 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-06T12:32:27.539 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.560 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.027) 0:07:13.952 ****** 2021-09-06T12:32:27.566 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-09-06T12:32:27.567 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.588 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.028) 0:07:13.981 ****** 2021-09-06T12:32:27.594 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-06T12:32:27.596 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.620 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.031) 0:07:14.012 ****** 2021-09-06T12:32:27.626 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-09-06T12:32:27.628 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-09-06T12:32:27.653 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.033) 0:07:14.046 ****** 2021-09-06T12:32:27.659 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T12:32:27.661 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.684 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.031) 0:07:14.077 ****** 2021-09-06T12:32:27.691 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-09-06T12:32:27.692 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.715 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.030) 0:07:14.107 ****** 2021-09-06T12:32:27.721 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-09-06T12:32:27.723 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.747 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.032) 0:07:14.140 ****** 2021-09-06T12:32:27.774 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T12:32:27.775 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:27.795 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.047) 0:07:14.188 ****** 2021-09-06T12:32:27.905 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-09-06T12:32:27.906 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-09-06T12:32:27.931 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:27 +0000 (0:00:00.136) 0:07:14.324 ****** 2021-09-06T12:32:28.010 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T12:32:28.012 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005791", "end": "2021-09-06 12:32:27.979431", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-06 12:32:27.973640", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:28.125 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-09-06T12:32:28.127 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-09-06T12:32:28.147 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.216) 0:07:14.540 ****** 2021-09-06T12:32:28.173 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-09-06T12:32:28.175 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.195 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.047) 0:07:14.587 ****** 2021-09-06T12:32:28.201 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T12:32:28.202 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003482", "end": "2021-09-06 12:32:28.161172", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-06 12:32:28.157690", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:28.223 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-09-06T12:32:28.224 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.234 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.235 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.245 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.267 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.072) 0:07:14.660 ****** 2021-09-06T12:32:28.304 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-09-06T12:32:28.306 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.316 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.317 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.341 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.073) 0:07:14.734 ****** 2021-09-06T12:32:28.367 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T12:32:28.369 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003168", "end": "2021-09-06 12:32:28.335375", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 12:32:28.332207", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} TASK [testnode : set_fact] ***************************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.393 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.051) 0:07:14.785 ****** 2021-09-06T12:32:28.419 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-09-06T12:32:28.421 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.442 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.049) 0:07:14.835 ****** 2021-09-06T12:32:28.469 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-09-06T12:32:28.470 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.492 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.049) 0:07:14.884 ****** 2021-09-06T12:32:28.518 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-09-06T12:32:28.520 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.541 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.048) 0:07:14.933 ****** 2021-09-06T12:32:28.547 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T12:32:28.548 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003445", "end": "2021-09-06 12:32:28.512712", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-06 12:32:28.509267", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:28.580 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.039) 0:07:14.972 ****** 2021-09-06T12:32:28.588 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-09-06T12:32:28.590 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.614 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.034) 0:07:15.007 ****** 2021-09-06T12:32:28.621 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-06T12:32:28.644 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.030) 0:07:15.037 ****** 2021-09-06T12:32:28.651 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-09-06T12:32:28.653 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.681 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.036) 0:07:15.074 ****** 2021-09-06T12:32:28.688 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-06T12:32:28.690 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.713 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.744 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.062) 0:07:15.136 ****** 2021-09-06T12:32:28.771 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:28.793 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:28 +0000 (0:00:00.049) 0:07:15.185 ****** 2021-09-06T12:32:29.018 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-09-06T12:32:29.020 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005782", "end": "2021-09-06 12:32:28.984403", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-09-06 12:32:28.978621", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:29.195 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003297", "end": "2021-09-06 12:32:29.163248", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-09-06 12:32:29.159951", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:29.362 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003110", "end": "2021-09-06 12:32:29.338324", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-09-06 12:32:29.335214", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:29.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003568", "end": "2021-09-06 12:32:29.511513", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-09-06 12:32:29.507945", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:32:29.562 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:29 +0000 (0:00:00.769) 0:07:15.955 ****** 2021-09-06T12:32:29.589 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-09-06T12:32:29.611 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:29 +0000 (0:00:00.049) 0:07:16.004 ****** 2021-09-06T12:32:29.638 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-09-06T12:32:29.640 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:29.705 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-09-06T12:32:29.723 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:29 +0000 (0:00:00.111) 0:07:16.115 ****** 2021-09-06T12:32:29.743 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:29 +0000 (0:00:00.020) 0:07:16.136 ****** 2021-09-06T12:32:30.389 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi041.front.sepia.ceph.com] 2021-09-06T12:32:30.419 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-09-06T12:32:30.445 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.701) 0:07:16.837 ****** 2021-09-06T12:32:30.487 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.042) 0:07:16.880 ****** 2021-09-06T12:32:30.494 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:30.496 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:30.518 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.030) 0:07:16.910 ****** 2021-09-06T12:32:30.526 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:30.550 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.032) 0:07:16.942 ****** 2021-09-06T12:32:30.558 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:30.559 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-06T12:32:30.582 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.031) 0:07:16.974 ****** 2021-09-06T12:32:30.589 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-09-06T12:32:30.614 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.032) 0:07:17.007 ****** 2021-09-06T12:32:30.623 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-09-06T12:32:30.625 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-06T12:32:30.649 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.034) 0:07:17.041 ****** 2021-09-06T12:32:30.657 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-09-06T12:32:30.683 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:30 +0000 (0:00:00.033) 0:07:17.075 ****** 2021-09-06T12:32:51.008 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-06T12:32:51.010 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-06T12:32:51.030 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:20.347) 0:07:37.422 ****** 2021-09-06T12:32:51.057 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:51.059 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-06T12:32:51.078 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.047) 0:07:37.470 ****** 2021-09-06T12:32:51.103 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:51.105 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:51.125 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.047) 0:07:37.518 ****** 2021-09-06T12:32:51.351 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-06T12:32:51.353 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:51.363 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-09-06T12:32:51.365 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.noarch", "Installed: podman-catatonit-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Installed: podman-3.2.3-0.10.module+el8.4.0+11989+6676f7ad.x86_64", "Removed: podman-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-catatonit-3.0.1-6.module+el8.4.0+10607+f4da7515.x86_64", "Removed: podman-docker-3.0.1-6.module+el8.4.0+10607+f4da7515.noarch"]} 2021-09-06T12:32:51.386 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.260) 0:07:37.778 ****** 2021-09-06T12:32:51.412 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:51.414 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-09-06T12:32:51.434 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.047) 0:07:37.826 ****** 2021-09-06T12:32:51.459 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-09-06T12:32:51.461 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:32:51.483 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.049) 0:07:37.875 ****** 2021-09-06T12:32:51.529 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-09-06T12:32:51.534 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:51.561 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.078) 0:07:37.954 ****** 2021-09-06T12:32:51.704 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:51.891 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-09-06T12:32:51.915 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:51 +0000 (0:00:00.353) 0:07:38.307 ****** 2021-09-06T12:32:52.045 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** ok: [smithi041.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T12:32:52.380 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T12:32:52.473 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T12:32:52.498 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:52 +0000 (0:00:00.583) 0:07:38.891 ****** 2021-09-06T12:32:52.816 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-09-06T12:32:52.840 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:32:52 +0000 (0:00:00.341) 0:07:39.232 ****** 2021-09-06T12:33:06.900 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:33:06.924 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:06 +0000 (0:00:14.084) 0:07:53.316 ****** 2021-09-06T12:33:07.349 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-09-06T12:33:07.374 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:07 +0000 (0:00:00.449) 0:07:53.766 ****** 2021-09-06T12:33:09.733 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-06T12:33:09.735 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-bhomsbwz-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-bhomsbwz-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-06T12:33:09.737 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-06T12:33:09.757 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:09 +0000 (0:00:02.382) 0:07:56.149 ****** 2021-09-06T12:33:09.783 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-06T12:33:09.785 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:09.807 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:09 +0000 (0:00:00.050) 0:07:56.200 ****** 2021-09-06T12:33:09.833 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-06T12:33:09.835 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:09.855 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:09 +0000 (0:00:00.047) 0:07:56.247 ****** 2021-09-06T12:33:09.975 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-09-06T12:33:09.977 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-rl6a7ldh-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-rl6a7ldh-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0:", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py in 2021-09-06T12:33:09.978 INFO:teuthology.task.ansible.out:stall for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-09-06T12:33:09.998 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:09 +0000 (0:00:00.143) 0:07:56.391 ****** 2021-09-06T12:33:10.025 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-09-06T12:33:10.026 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:10.045 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:10 +0000 (0:00:00.046) 0:07:56.438 ****** 2021-09-06T12:33:10.081 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-09-06T12:33:10.083 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:10.106 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:10 +0000 (0:00:00.060) 0:07:56.499 ****** 2021-09-06T12:33:10.215 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-06T12:33:10.217 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.164900", "end": "2021-09-06 12:33:10.194824", "rc": 0, "start": "2021-09-06 12:33:10.029924", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:33:10.239 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:10 +0000 (0:00:00.132) 0:07:56.631 ****** 2021-09-06T12:33:10.277 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-06T12:33:10.278 INFO:teuthology.task.ansible.out:skipping: [smithi041.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:10.473 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-09-06T12:33:10.475 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.174952", "end": "2021-09-06 12:33:10.456526", "rc": 0, "start": "2021-09-06 12:33:10.281574", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-09-06T12:33:10.494 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:10 +0000 (0:00:00.255) 0:07:56.887 ****** 2021-09-06T12:33:10.521 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-09-06T12:33:10.523 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-09-06T12:33:10.588 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-09-06T12:33:10.589 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-09-06T12:33:10.590 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched 2021-09-06T12:33:10.592 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-09-06T12:33:10.604 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Monday 06 September 2021 12:33:10 +0000 (0:00:00.109) 0:07:56.996 ****** 2021-09-06T12:33:11.269 INFO:teuthology.task.ansible.out:ok: [smithi041.front.sepia.ceph.com] 2021-09-06T12:33:11.288 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-09-06T12:33:11.305 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** 2021-09-06T12:33:11.307 INFO:teuthology.task.ansible.out:Monday 06 September 2021 12:33:11 +0000 (0:00:00.701) 0:07:57.698 ****** 2021-09-06T12:33:11.504 INFO:teuthology.task.ansible.out:changed: [smithi041.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T12:33:11.526 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-09-06T12:33:11.528 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* 2021-09-06T12:33:11.528 INFO:teuthology.task.ansible.out:smithi041.front.sepia.ceph.com : ok=142 changed=41 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 smithi052.front.sepia.ceph.com : ok=142 changed=41 unreachable=0 failed=0 skipped=178 rescued=0 ignored=4 Monday 06 September 2021 12:33:11 +0000 (0:00:00.222) 0:07:57.921 ****** =============================================================================== 2021-09-06T12:33:11.529 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 61.09s common : nrpe - Ensure SELinux policy is up to date -------------------- 52.58s users : Update authorized_keys using the keys repo --------------------- 37.98s testnode : Install packages -------------------------------------------- 31.01s 2021-09-06T12:33:11.529 INFO:teuthology.task.ansible.out:testnode : Ensure ceph packages are not present. ----------------------- 29.96s testnode : Ensure ceph dependency packages are not present. ------------ 27.73s container-host : Install container packages ---------------------------- 20.35s common : Register with subscription-manager. --------------------------- 18.00s common : nrpe - Remove SELinux policy package -------------------------- 14.66s common : nrpe - Load SELinux policy package ---------------------------- 14.39s container-host : Ensure git is installed ------------------------------- 14.08s common : Install Kerberos Packages (RedHat) ---------------------------- 12.83s testnode : Install epel packages --------------------------------------- 12.11s Remove revoked users ---------------------------------------------------- 9.81s testnode : Zap all non-root disks --------------------------------------- 5.88s container-host : Install registries-conf-ctl ---------------------------- 2.38s common : Enable necessary rhsm repos ------------------------------------ 1.99s users : Update authorized_keys for each user with literal keys ---------- 1.81s common : nrpe - Install SELinux dependencies ---------------------------- 1.66s Gathering Facts --------------------------------------------------------- 1.48s 2021-09-06T12:33:11.876 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi041.front.sepia.ceph.com'), Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2021-09-06T12:33:11.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T12:33:11.879 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T12:33:11.949 DEBUG:teuthology.orchestra.run.smithi041:> true 2021-09-06T12:33:12.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-06T12:33:12.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-09-06T12:33:12.086 DEBUG:teuthology.orchestra.run.smithi052:> true 2021-09-06T12:33:12.149 INFO:teuthology.run_tasks:Running task clock... 2021-09-06T12:33:12.163 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-09-06T12:33:12.164 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-06T12:33:12.165 DEBUG:teuthology.orchestra.run.smithi041:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-06T12:33:12.197 INFO:teuthology.orchestra.run.smithi041.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-06T12:33:12.212 INFO:teuthology.orchestra.run.smithi041.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-06T12:33:12.245 INFO:teuthology.orchestra.run.smithi041.stderr:sudo: ntpd: command not found 2021-09-06T12:33:12.255 INFO:teuthology.orchestra.run.smithi041.stdout:506 Cannot talk to daemon 2021-09-06T12:33:12.270 INFO:teuthology.orchestra.run.smithi041.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-06T12:33:12.284 INFO:teuthology.orchestra.run.smithi041.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-06T12:33:12.362 INFO:teuthology.orchestra.run.smithi041.stderr:bash: ntpq: command not found 2021-09-06T12:33:12.365 INFO:teuthology.orchestra.run.smithi041.stdout:210 Number of sources = 4 2021-09-06T12:33:12.365 INFO:teuthology.orchestra.run.smithi041.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-06T12:33:12.366 INFO:teuthology.orchestra.run.smithi041.stdout:=============================================================================== 2021-09-06T12:33:12.366 INFO:teuthology.orchestra.run.smithi041.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.367 INFO:teuthology.orchestra.run.smithi041.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.367 INFO:teuthology.orchestra.run.smithi041.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.367 INFO:teuthology.orchestra.run.smithi041.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.368 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-09-06T12:33:12.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-09-06T12:33:12.401 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-09-06T12:33:12.418 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-09-06T12:33:12.456 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2021-09-06T12:33:12.467 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2021-09-06T12:33:12.482 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-09-06T12:33:12.498 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-09-06T12:33:12.566 INFO:teuthology.orchestra.run.smithi052.stderr:bash: ntpq: command not found 2021-09-06T12:33:12.570 INFO:teuthology.orchestra.run.smithi052.stdout:210 Number of sources = 4 2021-09-06T12:33:12.570 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-09-06T12:33:12.570 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2021-09-06T12:33:12.571 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.571 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.571 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.571 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-09-06T12:33:12.572 INFO:teuthology.run_tasks:Running task install... 2021-09-06T12:33:12.583 DEBUG:teuthology.task.install:project ceph 2021-09-06T12:33:12.584 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'}} 2021-09-06T12:33:12.584 DEBUG:teuthology.task.install:config {'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'} 2021-09-06T12:33:12.585 INFO:teuthology.task.install:Using flavor: basic 2021-09-06T12:33:12.591 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-09-06T12:33:12.592 INFO:teuthology.task.install:extra packages: [] 2021-09-06T12:33:12.593 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-09-06T12:33:12.635 INFO:teuthology.orchestra.run.smithi041.stdout:RedHatEnterprise 2021-09-06T12:33:12.637 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:33:12.639 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-09-06T12:33:12.640 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'tag': None, 'wait_for_package': False} 2021-09-06T12:33:12.642 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-09-06T12:33:12.684 INFO:teuthology.orchestra.run.smithi052.stdout:RedHatEnterprise 2021-09-06T12:33:12.687 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:33:12.688 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'tag': None, 'wait_for_package': False} 2021-09-06T12:33:12.689 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-09-06T12:33:12.717 INFO:teuthology.orchestra.run.smithi041.stdout:RedHatEnterprise 2021-09-06T12:33:12.718 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:33:12.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0eb1a7943dd70e2a0b3086ea680284137a187e73 2021-09-06T12:33:12.765 INFO:teuthology.orchestra.run.smithi052.stdout:RedHatEnterprise 2021-09-06T12:33:12.766 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:33:12.767 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0eb1a7943dd70e2a0b3086ea680284137a187e73 2021-09-06T12:33:12.887 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/ 2021-09-06T12:33:12.888 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7472.g0eb1a794 2021-09-06T12:33:12.910 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/ 2021-09-06T12:33:12.910 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7472.g0eb1a794 2021-09-06T12:33:13.019 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-06T12:33:13.019 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:33:13.020 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-06T12:33:13.043 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-skoduri-lua/0eb1a7943dd70e2a0b3086ea680284137a187e73/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-09-06T12:33:13.044 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:33:13.044 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-09-06T12:33:13.046 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-06T12:33:13.046 DEBUG:teuthology.orchestra.run.smithi041:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0eb1a7943dd70e2a0b3086ea680284137a187e73/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-06T12:33:13.071 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-09-06T12:33:13.072 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0eb1a7943dd70e2a0b3086ea680284137a187e73/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-09-06T12:33:13.111 DEBUG:teuthology.orchestra.run.smithi041:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-06T12:33:13.138 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-09-06T12:33:13.187 DEBUG:teuthology.orchestra.run.smithi041:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-06T12:33:13.213 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-09-06T12:33:13.251 INFO:teuthology.orchestra.run.smithi041.stdout:check_obsoletes = 1 2021-09-06T12:33:13.252 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum clean all 2021-09-06T12:33:13.277 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2021-09-06T12:33:13.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2021-09-06T12:33:13.626 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:33:13.652 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:33:13.941 INFO:teuthology.orchestra.run.smithi041.stdout:62 files removed 2021-09-06T12:33:13.970 INFO:teuthology.orchestra.run.smithi052.stdout:62 files removed 2021-09-06T12:33:13.986 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-radosgw 2021-09-06T12:33:14.015 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2021-09-06T12:33:14.319 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:33:14.347 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:33:14.990 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 342 kB/s | 76 kB 00:00 2021-09-06T12:33:14.990 INFO:teuthology.orchestra.run.smithi041.stdout:ceph packages for x86_64 305 kB/s | 76 kB 00:00 2021-09-06T12:33:15.210 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 39 kB/s | 7.4 kB 00:00 2021-09-06T12:33:15.216 INFO:teuthology.orchestra.run.smithi041.stdout:ceph noarch packages 38 kB/s | 7.4 kB 00:00 2021-09-06T12:33:15.425 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 7.4 kB/s | 1.4 kB 00:00 2021-09-06T12:33:15.444 INFO:teuthology.orchestra.run.smithi041.stdout:ceph source packages 7.0 kB/s | 1.4 kB 00:00 2021-09-06T12:33:17.675 INFO:teuthology.orchestra.run.smithi052.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-09-06T12:33:17.713 INFO:teuthology.orchestra.run.smithi041.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2021-09-06T12:33:18.381 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 15 MB/s | 10 MB 00:00 2021-09-06T12:33:18.421 INFO:teuthology.orchestra.run.smithi041.stdout:Extra Packages for Enterprise Linux 15 MB/s | 10 MB 00:00 2021-09-06T12:33:20.506 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 3.5 MB/s | 24 kB 00:00 2021-09-06T12:33:20.541 INFO:teuthology.orchestra.run.smithi041.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2021-09-06T12:33:20.886 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 100 MB/s | 32 MB 00:00 2021-09-06T12:33:20.910 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 114 MB/s | 35 MB 00:00 2021-09-06T12:33:26.188 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 MB/s | 35 MB 00:00 2021-09-06T12:33:26.264 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 103 MB/s | 32 MB 00:00 2021-09-06T12:33:34.869 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:33:34.872 INFO:teuthology.orchestra.run.smithi052.stdout:==================================================================================================== 2021-09-06T12:33:34.872 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:33:34.872 INFO:teuthology.orchestra.run.smithi052.stdout:==================================================================================================== 2021-09-06T12:33:34.873 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:33:34.873 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 8.4 M 2021-09-06T12:33:34.873 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2021-09-06T12:33:34.873 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.4 M 2021-09-06T12:33:34.874 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.9 M 2021-09-06T12:33:34.874 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:33:34.874 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 5.2 M 2021-09-06T12:33:34.874 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 20 M 2021-09-06T12:33:34.874 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 24 k 2021-09-06T12:33:34.875 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-06T12:33:34.875 INFO:teuthology.orchestra.run.smithi052.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-06T12:33:34.875 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 785 k 2021-09-06T12:33:34.875 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-06T12:33:34.876 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-06T12:33:34.876 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 483 k 2021-09-06T12:33:34.876 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-06T12:33:34.876 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.7 M 2021-09-06T12:33:34.876 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-06T12:33:34.877 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-06T12:33:34.877 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 45 k 2021-09-06T12:33:34.877 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 78 k 2021-09-06T12:33:34.877 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 178 k 2021-09-06T12:33:34.878 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-06T12:33:34.878 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 365 k 2021-09-06T12:33:34.878 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 345 k 2021-09-06T12:33:34.878 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 109 k 2021-09-06T12:33:34.878 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:33:34.879 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:33:34.879 INFO:teuthology.orchestra.run.smithi052.stdout:==================================================================================================== 2021-09-06T12:33:34.879 INFO:teuthology.orchestra.run.smithi052.stdout:Install 21 Packages 2021-09-06T12:33:34.879 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2021-09-06T12:33:34.879 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:33:34.880 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 48 M 2021-09-06T12:33:34.880 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:33:34.914 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:33:34.917 INFO:teuthology.orchestra.run.smithi041.stdout:==================================================================================================== 2021-09-06T12:33:34.918 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:33:34.918 INFO:teuthology.orchestra.run.smithi041.stdout:==================================================================================================== 2021-09-06T12:33:34.918 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:33:34.918 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-radosgw x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 8.4 M 2021-09-06T12:33:34.919 INFO:teuthology.orchestra.run.smithi041.stdout:Upgrading: 2021-09-06T12:33:34.919 INFO:teuthology.orchestra.run.smithi041.stdout: librados2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.4 M 2021-09-06T12:33:34.919 INFO:teuthology.orchestra.run.smithi041.stdout: librbd1 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.9 M 2021-09-06T12:33:34.919 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:33:34.920 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-base x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 5.2 M 2021-09-06T12:33:34.920 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-common x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 20 M 2021-09-06T12:33:34.920 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-selinux x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 24 k 2021-09-06T12:33:34.920 INFO:teuthology.orchestra.run.smithi041.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-09-06T12:33:34.920 INFO:teuthology.orchestra.run.smithi041.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-09-06T12:33:34.921 INFO:teuthology.orchestra.run.smithi041.stdout: libcephfs2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 785 k 2021-09-06T12:33:34.921 INFO:teuthology.orchestra.run.smithi041.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-09-06T12:33:34.921 INFO:teuthology.orchestra.run.smithi041.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2021-09-06T12:33:34.921 INFO:teuthology.orchestra.run.smithi041.stdout: libradosstriper1 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 483 k 2021-09-06T12:33:34.922 INFO:teuthology.orchestra.run.smithi041.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2021-09-06T12:33:34.922 INFO:teuthology.orchestra.run.smithi041.stdout: librgw2 x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.7 M 2021-09-06T12:33:34.922 INFO:teuthology.orchestra.run.smithi041.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-09-06T12:33:34.922 INFO:teuthology.orchestra.run.smithi041.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2021-09-06T12:33:34.922 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-argparse x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 45 k 2021-09-06T12:33:34.923 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-common x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 78 k 2021-09-06T12:33:34.923 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cephfs x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 178 k 2021-09-06T12:33:34.923 INFO:teuthology.orchestra.run.smithi041.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2021-09-06T12:33:34.923 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rados x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 365 k 2021-09-06T12:33:34.924 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rbd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 345 k 2021-09-06T12:33:34.924 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rgw x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 109 k 2021-09-06T12:33:34.924 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:33:34.924 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:33:34.924 INFO:teuthology.orchestra.run.smithi041.stdout:==================================================================================================== 2021-09-06T12:33:34.925 INFO:teuthology.orchestra.run.smithi041.stdout:Install 21 Packages 2021-09-06T12:33:34.925 INFO:teuthology.orchestra.run.smithi041.stdout:Upgrade 2 Packages 2021-09-06T12:33:34.925 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:33:34.926 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 48 M 2021-09-06T12:33:34.926 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:33:36.528 INFO:teuthology.orchestra.run.smithi041.stdout:(1/23): ceph-base-17.0.0-7472.g0eb1a794.el8.x86 3.2 MB/s | 5.2 MB 00:01 2021-09-06T12:33:36.541 INFO:teuthology.orchestra.run.smithi052.stdout:(1/23): ceph-radosgw-17.0.0-7472.g0eb1a794.el8. 5.0 MB/s | 8.4 MB 00:01 2021-09-06T12:33:36.602 INFO:teuthology.orchestra.run.smithi052.stdout:(2/23): ceph-selinux-17.0.0-7472.g0eb1a794.el8. 403 kB/s | 24 kB 00:00 2021-09-06T12:33:36.615 INFO:teuthology.orchestra.run.smithi041.stdout:(2/23): ceph-selinux-17.0.0-7472.g0eb1a794.el8. 278 kB/s | 24 kB 00:00 2021-09-06T12:33:36.752 INFO:teuthology.orchestra.run.smithi052.stdout:(3/23): libcephfs2-17.0.0-7472.g0eb1a794.el8.x8 5.1 MB/s | 785 kB 00:00 2021-09-06T12:33:36.773 INFO:teuthology.orchestra.run.smithi052.stdout:(4/23): ceph-base-17.0.0-7472.g0eb1a794.el8.x86 2.7 MB/s | 5.2 MB 00:01 2021-09-06T12:33:36.900 INFO:teuthology.orchestra.run.smithi041.stdout:(3/23): libcephfs2-17.0.0-7472.g0eb1a794.el8.x8 2.7 MB/s | 785 kB 00:00 2021-09-06T12:33:36.921 INFO:teuthology.orchestra.run.smithi052.stdout:(5/23): ceph-common-17.0.0-7472.g0eb1a794.el8.x 9.6 MB/s | 20 MB 00:02 2021-09-06T12:33:36.925 INFO:teuthology.orchestra.run.smithi052.stdout:(6/23): libradosstriper1-17.0.0-7472.g0eb1a794. 2.7 MB/s | 483 kB 00:00 2021-09-06T12:33:36.954 INFO:teuthology.orchestra.run.smithi052.stdout:(7/23): python3-ceph-argparse-17.0.0-7472.g0eb1 1.3 MB/s | 45 kB 00:00 2021-09-06T12:33:36.962 INFO:teuthology.orchestra.run.smithi052.stdout:(8/23): python3-ceph-common-17.0.0-7472.g0eb1a7 2.0 MB/s | 78 kB 00:00 2021-09-06T12:33:37.000 INFO:teuthology.orchestra.run.smithi052.stdout:(9/23): python3-cephfs-17.0.0-7472.g0eb1a794.el 3.8 MB/s | 178 kB 00:00 2021-09-06T12:33:37.018 INFO:teuthology.orchestra.run.smithi052.stdout:(10/23): python3-rados-17.0.0-7472.g0eb1a794.el 6.4 MB/s | 365 kB 00:00 2021-09-06T12:33:37.032 INFO:teuthology.orchestra.run.smithi041.stdout:(4/23): libradosstriper1-17.0.0-7472.g0eb1a794. 3.6 MB/s | 483 kB 00:00 2021-09-06T12:33:37.059 INFO:teuthology.orchestra.run.smithi052.stdout:(11/23): python3-rbd-17.0.0-7472.g0eb1a794.el8. 5.8 MB/s | 345 kB 00:00 2021-09-06T12:33:37.067 INFO:teuthology.orchestra.run.smithi052.stdout:(12/23): python3-rgw-17.0.0-7472.g0eb1a794.el8. 2.2 MB/s | 109 kB 00:00 2021-09-06T12:33:37.131 INFO:teuthology.orchestra.run.smithi041.stdout:(5/23): ceph-radosgw-17.0.0-7472.g0eb1a794.el8. 3.8 MB/s | 8.4 MB 00:02 2021-09-06T12:33:37.159 INFO:teuthology.orchestra.run.smithi041.stdout:(6/23): python3-ceph-argparse-17.0.0-7472.g0eb1 1.6 MB/s | 45 kB 00:00 2021-09-06T12:33:37.184 INFO:teuthology.orchestra.run.smithi041.stdout:(7/23): python3-ceph-common-17.0.0-7472.g0eb1a7 3.1 MB/s | 78 kB 00:00 2021-09-06T12:33:37.208 INFO:teuthology.orchestra.run.smithi052.stdout:(13/23): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2021-09-06T12:33:37.209 INFO:teuthology.orchestra.run.smithi052.stdout:(14/23): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2021-09-06T12:33:37.213 INFO:teuthology.orchestra.run.smithi041.stdout:(8/23): python3-cephfs-17.0.0-7472.g0eb1a794.el 6.0 MB/s | 178 kB 00:00 2021-09-06T12:33:37.234 INFO:teuthology.orchestra.run.smithi052.stdout:(15/23): liboath-2.6.2-3.el8.x86_64.rpm 2.3 MB/s | 59 kB 00:00 2021-09-06T12:33:37.246 INFO:teuthology.orchestra.run.smithi052.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2021-09-06T12:33:37.252 INFO:teuthology.orchestra.run.smithi041.stdout:(9/23): python3-rados-17.0.0-7472.g0eb1a794.el8 9.4 MB/s | 365 kB 00:00 2021-09-06T12:33:37.283 INFO:teuthology.orchestra.run.smithi052.stdout:(17/23): librdkafka-0.11.4-1.el8.x86_64.rpm 7.0 MB/s | 353 kB 00:00 2021-09-06T12:33:37.289 INFO:teuthology.orchestra.run.smithi041.stdout:(10/23): python3-rbd-17.0.0-7472.g0eb1a794.el8. 9.2 MB/s | 345 kB 00:00 2021-09-06T12:33:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.5 MB/s | 259 kB 00:00 2021-09-06T12:33:37.318 INFO:teuthology.orchestra.run.smithi041.stdout:(11/23): python3-rgw-17.0.0-7472.g0eb1a794.el8. 3.7 MB/s | 109 kB 00:00 2021-09-06T12:33:37.319 INFO:teuthology.orchestra.run.smithi052.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.5 MB/s | 44 kB 00:00 2021-09-06T12:33:37.334 INFO:teuthology.orchestra.run.smithi052.stdout:(20/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.9 MB/s | 47 kB 00:00 2021-09-06T12:33:37.446 INFO:teuthology.orchestra.run.smithi052.stdout:(21/23): librgw2-17.0.0-7472.g0eb1a794.el8.x86_ 5.5 MB/s | 3.7 MB 00:00 2021-09-06T12:33:37.485 INFO:teuthology.orchestra.run.smithi041.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.8 MB/s | 306 kB 00:00 2021-09-06T12:33:37.515 INFO:teuthology.orchestra.run.smithi041.stdout:(13/23): leveldb-1.22-1.el8.x86_64.rpm 5.9 MB/s | 181 kB 00:00 2021-09-06T12:33:37.544 INFO:teuthology.orchestra.run.smithi041.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 2.0 MB/s | 59 kB 00:00 2021-09-06T12:33:37.573 INFO:teuthology.orchestra.run.smithi041.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 2.6 MB/s | 75 kB 00:00 2021-09-06T12:33:37.611 INFO:teuthology.orchestra.run.smithi041.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2021-09-06T12:33:37.640 INFO:teuthology.orchestra.run.smithi041.stdout:(17/23): librdkafka-0.11.4-1.el8.x86_64.rpm 12 MB/s | 353 kB 00:00 2021-09-06T12:33:37.645 INFO:teuthology.orchestra.run.smithi052.stdout:(22/23): librados2-17.0.0-7472.g0eb1a794.el8.x8 10 MB/s | 3.4 MB 00:00 2021-09-06T12:33:37.666 INFO:teuthology.orchestra.run.smithi041.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 9.6 MB/s | 259 kB 00:00 2021-09-06T12:33:37.690 INFO:teuthology.orchestra.run.smithi041.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 1.8 MB/s | 44 kB 00:00 2021-09-06T12:33:37.728 INFO:teuthology.orchestra.run.smithi041.stdout:(20/23): librgw2-17.0.0-7472.g0eb1a794.el8.x86_ 5.4 MB/s | 3.7 MB 00:00 2021-09-06T12:33:37.820 INFO:teuthology.orchestra.run.smithi052.stdout:(23/23): librbd1-17.0.0-7472.g0eb1a794.el8.x86_ 8.0 MB/s | 3.9 MB 00:00 2021-09-06T12:33:37.823 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:33:37.823 INFO:teuthology.orchestra.run.smithi052.stdout:Total 16 MB/s | 48 MB 00:02 2021-09-06T12:33:37.828 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:33:37.878 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:33:37.878 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:33:37.962 INFO:teuthology.orchestra.run.smithi041.stdout:(21/23): librados2-17.0.0-7472.g0eb1a794.el8.x8 13 MB/s | 3.4 MB 00:00 2021-09-06T12:33:38.169 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:33:38.173 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:33:38.197 INFO:teuthology.orchestra.run.smithi041.stdout:(22/23): librbd1-17.0.0-7472.g0eb1a794.el8.x86_ 8.2 MB/s | 3.9 MB 00:00 2021-09-06T12:33:38.638 INFO:teuthology.orchestra.run.smithi041.stdout:(23/23): ceph-common-17.0.0-7472.g0eb1a794.el8. 5.3 MB/s | 20 MB 00:03 2021-09-06T12:33:38.642 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:33:38.643 INFO:teuthology.orchestra.run.smithi041.stdout:Total 13 MB/s | 48 MB 00:03 2021-09-06T12:33:38.647 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:33:38.697 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:33:38.697 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:33:38.981 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:33:38.986 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:33:39.854 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:33:39.994 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-06T12:33:40.185 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:33:40.426 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-06T12:33:40.464 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-06T12:33:40.635 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:33:40.737 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:33:40.874 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 1/25 2021-09-06T12:33:40.924 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/25 2021-09-06T12:33:41.014 INFO:teuthology.orchestra.run.smithi041.stdout: Upgrading : librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:33:41.120 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:33:41.268 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-06T12:33:41.270 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/25 2021-09-06T12:33:41.385 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:33:41.517 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : liboath-2.6.2-3.el8.x86_64 4/25 2021-09-06T12:33:41.831 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:33:41.846 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:33:42.060 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/25 2021-09-06T12:33:42.142 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/25 2021-09-06T12:33:42.279 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:33:42.328 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-09-06T12:33:42.476 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T12:33:42.480 INFO:teuthology.orchestra.run.smithi041.stdout: Upgrading : librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/25 2021-09-06T12:33:42.577 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/25 2021-09-06T12:33:42.722 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 7/25 2021-09-06T12:33:42.790 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 7/25 2021-09-06T12:33:42.896 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-09-06T12:33:43.052 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 9/25 2021-09-06T12:33:43.132 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 9/25 2021-09-06T12:33:43.265 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/25 2021-09-06T12:33:43.292 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-06T12:33:43.430 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 9/25 2021-09-06T12:33:43.433 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8. 11/25 2021-09-06T12:33:43.513 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 9/25 2021-09-06T12:33:43.597 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 12/25 2021-09-06T12:33:43.687 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-09-06T12:33:43.821 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 13/25 2021-09-06T12:33:43.830 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8. 11/25 2021-09-06T12:33:44.010 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 12/25 2021-09-06T12:33:44.056 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 14/25 2021-09-06T12:33:44.192 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 13/25 2021-09-06T12:33:44.226 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 15/25 2021-09-06T12:33:44.304 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 15/25 2021-09-06T12:33:44.418 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 14/25 2021-09-06T12:33:44.471 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-06T12:33:44.539 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 15/25 2021-09-06T12:33:44.625 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 15/25 2021-09-06T12:33:44.664 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-06T12:33:44.766 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-09-06T12:33:44.870 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-06T12:33:44.959 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-06T12:33:45.043 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x8 19/25 2021-09-06T12:33:45.158 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-09-06T12:33:45.314 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x8 19/25 2021-09-06T12:33:49.444 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:49.651 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:49.863 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:50.047 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:50.255 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:50.520 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 21/25 2021-09-06T12:33:50.632 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 21/25 2021-09-06T12:33:50.655 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 20/25 2021-09-06T12:33:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:33:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 21/25 2021-09-06T12:33:51.022 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 21/25 2021-09-06T12:33:51.121 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:34:09.373 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:34:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2021-09-06T12:34:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2021-09-06T12:34:09.373 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /dev 2021-09-06T12:34:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /run 2021-09-06T12:34:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2021-09-06T12:34:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2021-09-06T12:34:09.374 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2021-09-06T12:34:09.375 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2021-09-06T12:34:09.375 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /dev 2021-09-06T12:34:09.375 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:34:09.668 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:34:09.669 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /sys 2021-09-06T12:34:09.669 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /proc 2021-09-06T12:34:09.669 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /dev 2021-09-06T12:34:09.669 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /run 2021-09-06T12:34:09.670 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /mnt 2021-09-06T12:34:09.670 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /var/tmp 2021-09-06T12:34:09.670 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /home 2021-09-06T12:34:09.670 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /tmp 2021-09-06T12:34:09.670 INFO:teuthology.orchestra.run.smithi041.stdout:skipping the directory /dev 2021-09-06T12:34:09.671 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:34:10.558 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 23/25 2021-09-06T12:34:10.640 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 23/25 2021-09-06T12:34:10.641 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T12:34:10.808 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 23/25 2021-09-06T12:34:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T12:34:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:10.886 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 23/25 2021-09-06T12:34:10.887 INFO:teuthology.orchestra.run.smithi041.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T12:34:11.121 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-09-06T12:34:11.121 INFO:teuthology.orchestra.run.smithi041.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:11.604 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/25 2021-09-06T12:34:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:34:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/25 2021-09-06T12:34:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 4/25 2021-09-06T12:34:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:34:11.605 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 6/25 2021-09-06T12:34:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 7/25 2021-09-06T12:34:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8. 8/25 2021-09-06T12:34:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x8 9/25 2021-09-06T12:34:11.606 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 10/25 2021-09-06T12:34:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 11/25 2021-09-06T12:34:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 12/25 2021-09-06T12:34:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 13/25 2021-09-06T12:34:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-06T12:34:11.607 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-06T12:34:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-06T12:34:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-06T12:34:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 18/25 2021-09-06T12:34:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2021-09-06T12:34:11.608 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2021-09-06T12:34:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 21/25 2021-09-06T12:34:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:34:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-06T12:34:11.609 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 24/25 2021-09-06T12:34:11.896 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:11.896 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/25 2021-09-06T12:34:11.897 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/25 2021-09-06T12:34:11.897 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/25 2021-09-06T12:34:11.897 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 4/25 2021-09-06T12:34:11.897 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/25 2021-09-06T12:34:11.898 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_6 6/25 2021-09-06T12:34:11.898 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 7/25 2021-09-06T12:34:11.898 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8. 8/25 2021-09-06T12:34:11.898 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x8 9/25 2021-09-06T12:34:11.898 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 10/25 2021-09-06T12:34:11.899 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 11/25 2021-09-06T12:34:11.899 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 12/25 2021-09-06T12:34:11.899 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 13/25 2021-09-06T12:34:11.899 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2021-09-06T12:34:11.899 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : leveldb-1.22-1.el8.x86_64 15/25 2021-09-06T12:34:11.900 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 16/25 2021-09-06T12:34:11.900 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 17/25 2021-09-06T12:34:11.900 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2021-09-06T12:34:11.900 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 19/25 2021-09-06T12:34:11.901 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 20/25 2021-09-06T12:34:11.901 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 21/25 2021-09-06T12:34:11.901 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 22/25 2021-09-06T12:34:11.901 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-09-06T12:34:11.901 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 24/25 2021-09-06T12:34:12.681 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:12.681 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:34:12.744 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-09-06T12:34:12.744 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:34:23.859 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:18 PM UTC. 2021-09-06T12:34:24.029 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:18 PM UTC. 2021-09-06T12:34:26.410 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:34:26.411 INFO:teuthology.orchestra.run.smithi041.stdout:Upgraded: 2021-09-06T12:34:26.411 INFO:teuthology.orchestra.run.smithi041.stdout: librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.411 INFO:teuthology.orchestra.run.smithi041.stdout: librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.411 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:34:26.412 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.412 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.412 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.412 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.412 INFO:teuthology.orchestra.run.smithi041.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-06T12:34:26.413 INFO:teuthology.orchestra.run.smithi041.stdout: leveldb-1.22-1.el8.x86_64 2021-09-06T12:34:26.413 INFO:teuthology.orchestra.run.smithi041.stdout: libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.413 INFO:teuthology.orchestra.run.smithi041.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-06T12:34:26.413 INFO:teuthology.orchestra.run.smithi041.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-06T12:34:26.413 INFO:teuthology.orchestra.run.smithi041.stdout: libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.414 INFO:teuthology.orchestra.run.smithi041.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-06T12:34:26.414 INFO:teuthology.orchestra.run.smithi041.stdout: librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.414 INFO:teuthology.orchestra.run.smithi041.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-06T12:34:26.414 INFO:teuthology.orchestra.run.smithi041.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-06T12:34:26.414 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.415 INFO:teuthology.orchestra.run.smithi041.stdout: python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.415 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.415 INFO:teuthology.orchestra.run.smithi041.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-06T12:34:26.415 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.416 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.416 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.416 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:34:26.416 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:34:26.570 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-test 2021-09-06T12:34:26.681 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:34:26.681 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2021-09-06T12:34:26.681 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.681 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.682 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:34:26.682 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.682 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.682 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.683 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.683 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-09-06T12:34:26.683 INFO:teuthology.orchestra.run.smithi052.stdout: leveldb-1.22-1.el8.x86_64 2021-09-06T12:34:26.683 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.683 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.2-3.el8.x86_64 2021-09-06T12:34:26.684 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-09-06T12:34:26.684 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.684 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-09-06T12:34:26.684 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.684 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.3.1-3.el8.x86_64 2021-09-06T12:34:26.685 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-09-06T12:34:26.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.685 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-09-06T12:34:26.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.686 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:34:26.686 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:34:26.686 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:34:26.816 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2021-09-06T12:34:26.917 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:34:27.163 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:34:27.437 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-06T12:34:27.680 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:34:32.739 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:34:32.906 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:34:37.850 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:34:38.374 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:34:40.487 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:34:40.488 INFO:teuthology.orchestra.run.smithi041.stdout:============================================================================================ 2021-09-06T12:34:40.488 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:34:40.488 INFO:teuthology.orchestra.run.smithi041.stdout:============================================================================================ 2021-09-06T12:34:40.488 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:34:40.488 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-test x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 43 M 2021-09-06T12:34:40.489 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:34:40.489 INFO:teuthology.orchestra.run.smithi041.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-06T12:34:40.489 INFO:teuthology.orchestra.run.smithi041.stdout: libcephsqlite x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 171 k 2021-09-06T12:34:40.489 INFO:teuthology.orchestra.run.smithi041.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-06T12:34:40.489 INFO:teuthology.orchestra.run.smithi041.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-06T12:34:40.490 INFO:teuthology.orchestra.run.smithi041.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-06T12:34:40.490 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:34:40.490 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:34:40.490 INFO:teuthology.orchestra.run.smithi041.stdout:============================================================================================ 2021-09-06T12:34:40.491 INFO:teuthology.orchestra.run.smithi041.stdout:Install 6 Packages 2021-09-06T12:34:40.491 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:34:40.491 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 44 M 2021-09-06T12:34:40.492 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 206 M 2021-09-06T12:34:40.492 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:34:40.642 INFO:teuthology.orchestra.run.smithi041.stdout:(1/6): libcephsqlite-17.0.0-7472.g0eb1a794.el8. 1.1 MB/s | 171 kB 00:00 2021-09-06T12:34:40.674 INFO:teuthology.orchestra.run.smithi041.stdout:(2/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 376 kB/s | 69 kB 00:00 2021-09-06T12:34:40.686 INFO:teuthology.orchestra.run.smithi041.stdout:(3/6): jq-1.5-12.el8.x86_64.rpm 3.6 MB/s | 161 kB 00:00 2021-09-06T12:34:40.718 INFO:teuthology.orchestra.run.smithi041.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.2 MB/s | 187 kB 00:00 2021-09-06T12:34:40.725 INFO:teuthology.orchestra.run.smithi041.stdout:(5/6): socat-1.7.3.3-2.el8.x86_64.rpm 7.5 MB/s | 302 kB 00:00 2021-09-06T12:34:41.045 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:34:41.045 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2021-09-06T12:34:41.046 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:34:41.046 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2021-09-06T12:34:41.046 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:34:41.046 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 43 M 2021-09-06T12:34:41.047 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:34:41.047 INFO:teuthology.orchestra.run.smithi052.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2021-09-06T12:34:41.047 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 171 k 2021-09-06T12:34:41.047 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2021-09-06T12:34:41.047 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2021-09-06T12:34:41.048 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-09-06T12:34:41.048 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:34:41.048 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:34:41.048 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================================ 2021-09-06T12:34:41.049 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2021-09-06T12:34:41.049 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:34:41.049 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 44 M 2021-09-06T12:34:41.050 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 206 M 2021-09-06T12:34:41.050 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:34:41.243 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 355 kB/s | 69 kB 00:00 2021-09-06T12:34:41.283 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): jq-1.5-12.el8.x86_64.rpm 3.9 MB/s | 161 kB 00:00 2021-09-06T12:34:41.309 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 7.0 MB/s | 187 kB 00:00 2021-09-06T12:34:41.338 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2021-09-06T12:34:41.374 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): libcephsqlite-17.0.0-7472.g0eb1a794.el8. 523 kB/s | 171 kB 00:00 2021-09-06T12:34:43.902 INFO:teuthology.orchestra.run.smithi041.stdout:(6/6): ceph-test-17.0.0-7472.g0eb1a794.el8.x86_ 13 MB/s | 43 MB 00:03 2021-09-06T12:34:43.905 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:34:43.905 INFO:teuthology.orchestra.run.smithi041.stdout:Total 13 MB/s | 44 MB 00:03 2021-09-06T12:34:43.909 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:34:43.935 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:34:43.935 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:34:44.189 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:34:44.194 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:34:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:34:45.746 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-06T12:34:45.889 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-06T12:34:45.983 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-06T12:34:46.160 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-06T12:34:46.297 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-06T12:34:46.447 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/6 2021-09-06T12:34:47.196 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-test-17.0.0-7472.g0eb1a794.el8.x86_ 7.0 MB/s | 43 MB 00:06 2021-09-06T12:34:47.200 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:34:47.200 INFO:teuthology.orchestra.run.smithi052.stdout:Total 7.2 MB/s | 44 MB 00:06 2021-09-06T12:34:47.204 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:34:47.229 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:34:47.229 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:34:47.482 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:34:47.491 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:34:47.970 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:34:48.139 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 1/6 2021-09-06T12:34:48.327 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-06T12:34:48.408 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 2/6 2021-09-06T12:34:48.588 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : jq-1.5-12.el8.x86_64 3/6 2021-09-06T12:34:48.709 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 4/6 2021-09-06T12:34:48.835 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/6 2021-09-06T12:34:50.964 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/6 2021-09-06T12:34:51.444 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/6 2021-09-06T12:34:51.681 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/6 2021-09-06T12:34:51.681 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/6 2021-09-06T12:34:51.681 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/6 2021-09-06T12:34:51.681 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-06T12:34:51.682 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-06T12:34:51.682 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-06T12:34:52.167 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-06T12:34:52.167 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:34:53.348 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/6 2021-09-06T12:34:53.598 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/6 2021-09-06T12:34:53.817 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 6/6 2021-09-06T12:34:53.817 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/6 2021-09-06T12:34:53.817 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/6 2021-09-06T12:34:53.817 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 3/6 2021-09-06T12:34:53.818 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : jq-1.5-12.el8.x86_64 4/6 2021-09-06T12:34:53.818 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2021-09-06T12:34:54.188 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 6/6 2021-09-06T12:34:54.189 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:35:03.097 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:18 PM UTC. 2021-09-06T12:35:05.284 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:18 PM UTC. 2021-09-06T12:35:05.644 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:05.645 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:35:05.645 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:05.645 INFO:teuthology.orchestra.run.smithi041.stdout: jq-1.5-12.el8.x86_64 2021-09-06T12:35:05.645 INFO:teuthology.orchestra.run.smithi041.stdout: libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:05.646 INFO:teuthology.orchestra.run.smithi041.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-06T12:35:05.646 INFO:teuthology.orchestra.run.smithi041.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-06T12:35:05.646 INFO:teuthology.orchestra.run.smithi041.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-06T12:35:05.646 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:05.647 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:35:05.774 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph 2021-09-06T12:35:06.112 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:35:06.657 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:35:07.868 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:07.868 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:35:07.868 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:07.869 INFO:teuthology.orchestra.run.smithi052.stdout: jq-1.5-12.el8.x86_64 2021-09-06T12:35:07.869 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:07.869 INFO:teuthology.orchestra.run.smithi052.stdout: oniguruma-6.8.2-2.el8.x86_64 2021-09-06T12:35:07.870 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.3.3-2.el8.x86_64 2021-09-06T12:35:07.870 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-09-06T12:35:07.870 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:07.870 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:35:07.996 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2021-09-06T12:35:08.335 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:35:08.856 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-06T12:35:11.867 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:35:13.944 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:35:17.026 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:35:19.314 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:35:19.669 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:35:19.673 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================= 2021-09-06T12:35:19.673 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:35:19.673 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================= 2021-09-06T12:35:19.673 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:35:19.674 INFO:teuthology.orchestra.run.smithi041.stdout: ceph x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 6.4 k 2021-09-06T12:35:19.674 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:35:19.674 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mds x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 2.3 M 2021-09-06T12:35:19.674 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 1.5 M 2021-09-06T12:35:19.675 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 214 k 2021-09-06T12:35:19.675 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mon x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.9 M 2021-09-06T12:35:19.675 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-osd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 17 M 2021-09-06T12:35:19.675 INFO:teuthology.orchestra.run.smithi041.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-06T12:35:19.675 INFO:teuthology.orchestra.run.smithi041.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-06T12:35:19.676 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-06T12:35:19.676 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-06T12:35:19.676 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-06T12:35:19.676 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-06T12:35:19.676 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-06T12:35:19.677 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-06T12:35:19.677 INFO:teuthology.orchestra.run.smithi041.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-06T12:35:19.677 INFO:teuthology.orchestra.run.smithi041.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-06T12:35:19.677 INFO:teuthology.orchestra.run.smithi041.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-06T12:35:19.678 INFO:teuthology.orchestra.run.smithi041.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-06T12:35:19.678 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-06T12:35:19.678 INFO:teuthology.orchestra.run.smithi041.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-06T12:35:19.678 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-06T12:35:19.678 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-06T12:35:19.679 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-06T12:35:19.679 INFO:teuthology.orchestra.run.smithi041.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-06T12:35:19.679 INFO:teuthology.orchestra.run.smithi041.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-06T12:35:19.679 INFO:teuthology.orchestra.run.smithi041.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-06T12:35:19.680 INFO:teuthology.orchestra.run.smithi041.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-06T12:35:19.680 INFO:teuthology.orchestra.run.smithi041.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-06T12:35:19.680 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-06T12:35:19.680 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-06T12:35:19.680 INFO:teuthology.orchestra.run.smithi041.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-06T12:35:19.681 INFO:teuthology.orchestra.run.smithi041.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-06T12:35:19.681 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:19.681 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:35:19.681 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================= 2021-09-06T12:35:19.681 INFO:teuthology.orchestra.run.smithi041.stdout:Install 32 Packages 2021-09-06T12:35:19.682 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:19.682 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 28 M 2021-09-06T12:35:19.683 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 113 M 2021-09-06T12:35:19.683 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:35:19.900 INFO:teuthology.orchestra.run.smithi041.stdout:(1/32): ceph-17.0.0-7472.g0eb1a794.el8.x86_64.r 29 kB/s | 6.4 kB 00:00 2021-09-06T12:35:20.839 INFO:teuthology.orchestra.run.smithi041.stdout:(2/32): ceph-mon-17.0.0-7472.g0eb1a794.el8.x86_ 4.1 MB/s | 3.9 MB 00:00 2021-09-06T12:35:21.088 INFO:teuthology.orchestra.run.smithi041.stdout:(3/32): ceph-mds-17.0.0-7472.g0eb1a794.el8.x86_ 1.6 MB/s | 2.3 MB 00:01 2021-09-06T12:35:21.196 INFO:teuthology.orchestra.run.smithi041.stdout:(4/32): ceph-mgr-17.0.0-7472.g0eb1a794.el8.x86_ 1.0 MB/s | 1.5 MB 00:01 2021-09-06T12:35:21.197 INFO:teuthology.orchestra.run.smithi041.stdout:(5/32): ceph-mgr-modules-core-17.0.0-7472.g0eb1 1.9 MB/s | 214 kB 00:00 2021-09-06T12:35:21.373 INFO:teuthology.orchestra.run.smithi041.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 246 kB/s | 44 kB 00:00 2021-09-06T12:35:21.467 INFO:teuthology.orchestra.run.smithi041.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 688 kB/s | 185 kB 00:00 2021-09-06T12:35:21.517 INFO:teuthology.orchestra.run.smithi041.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.2 MB/s | 173 kB 00:00 2021-09-06T12:35:21.566 INFO:teuthology.orchestra.run.smithi041.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 228 kB/s | 11 kB 00:00 2021-09-06T12:35:21.617 INFO:teuthology.orchestra.run.smithi041.stdout:(10/32): python3-jaraco-functools-2.0-4.el8.noa 341 kB/s | 18 kB 00:00 2021-09-06T12:35:21.669 INFO:teuthology.orchestra.run.smithi041.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 957 kB/s | 49 kB 00:00 2021-09-06T12:35:21.707 INFO:teuthology.orchestra.run.smithi041.stdout:(12/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2021-09-06T12:35:21.712 INFO:teuthology.orchestra.run.smithi041.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 1.3 MB/s | 59 kB 00:00 2021-09-06T12:35:21.761 INFO:teuthology.orchestra.run.smithi041.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 321 kB/s | 16 kB 00:00 2021-09-06T12:35:21.815 INFO:teuthology.orchestra.run.smithi041.stdout:(15/32): python3-simplegeneric-0.8.1-17.el8.noa 348 kB/s | 19 kB 00:00 2021-09-06T12:35:21.858 INFO:teuthology.orchestra.run.smithi041.stdout:(16/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2021-09-06T12:35:21.866 INFO:teuthology.orchestra.run.smithi041.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 465 kB/s | 24 kB 00:00 2021-09-06T12:35:21.913 INFO:teuthology.orchestra.run.smithi041.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 535 kB/s | 29 kB 00:00 2021-09-06T12:35:21.921 INFO:teuthology.orchestra.run.smithi041.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 501 kB/s | 27 kB 00:00 2021-09-06T12:35:22.018 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:35:22.021 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================= 2021-09-06T12:35:22.021 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:35:22.022 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================= 2021-09-06T12:35:22.022 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:35:22.022 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 6.4 k 2021-09-06T12:35:22.022 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:35:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 2.3 M 2021-09-06T12:35:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 1.5 M 2021-09-06T12:35:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 214 k 2021-09-06T12:35:22.023 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.9 M 2021-09-06T12:35:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 17 M 2021-09-06T12:35:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-09-06T12:35:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-09-06T12:35:22.024 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2021-09-06T12:35:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-09-06T12:35:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-09-06T12:35:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 3.2.1-4.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2021-09-06T12:35:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-09-06T12:35:22.025 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-09-06T12:35:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-09-06T12:35:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2021-09-06T12:35:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2021-09-06T12:35:22.026 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-09-06T12:35:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-09-06T12:35:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-09-06T12:35:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2021-09-06T12:35:22.027 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2021-09-06T12:35:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2021-09-06T12:35:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-09-06T12:35:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-09-06T12:35:22.028 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-09-06T12:35:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-09-06T12:35:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-09-06T12:35:22.029 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-09-06T12:35:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-09-06T12:35:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2021-09-06T12:35:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-09-06T12:35:22.030 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:22.030 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:35:22.031 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================= 2021-09-06T12:35:22.031 INFO:teuthology.orchestra.run.smithi052.stdout:Install 32 Packages 2021-09-06T12:35:22.031 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:22.032 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 28 M 2021-09-06T12:35:22.032 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 113 M 2021-09-06T12:35:22.033 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:35:22.128 INFO:teuthology.orchestra.run.smithi041.stdout:(20/32): ceph-osd-17.0.0-7472.g0eb1a794.el8.x86 13 MB/s | 17 MB 00:01 2021-09-06T12:35:22.129 INFO:teuthology.orchestra.run.smithi041.stdout:(21/32): python3-waitress-1.2.1-2.el8.1.noarch. 1.1 MB/s | 241 kB 00:00 2021-09-06T12:35:22.130 INFO:teuthology.orchestra.run.smithi041.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2021-09-06T12:35:22.160 INFO:teuthology.orchestra.run.smithi052.stdout:(1/32): ceph-17.0.0-7472.g0eb1a794.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2021-09-06T12:35:22.172 INFO:teuthology.orchestra.run.smithi041.stdout:(23/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.7 MB/s | 238 kB 00:00 2021-09-06T12:35:22.181 INFO:teuthology.orchestra.run.smithi041.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 446 kB/s | 23 kB 00:00 2021-09-06T12:35:22.200 INFO:teuthology.orchestra.run.smithi041.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 3.9 MB/s | 109 kB 00:00 2021-09-06T12:35:22.236 INFO:teuthology.orchestra.run.smithi041.stdout:(26/32): python3-cryptography-3.2.1-4.el8.x86_6 10 MB/s | 559 kB 00:00 2021-09-06T12:35:22.247 INFO:teuthology.orchestra.run.smithi041.stdout:(27/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.3 MB/s | 157 kB 00:00 2021-09-06T12:35:22.262 INFO:teuthology.orchestra.run.smithi041.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.0 MB/s | 54 kB 00:00 2021-09-06T12:35:22.278 INFO:teuthology.orchestra.run.smithi041.stdout:(29/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 3.2 MB/s | 103 kB 00:00 2021-09-06T12:35:22.305 INFO:teuthology.orchestra.run.smithi041.stdout:(30/32): python3-werkzeug-0.12.2-4.el8.noarch.r 11 MB/s | 457 kB 00:00 2021-09-06T12:35:22.309 INFO:teuthology.orchestra.run.smithi041.stdout:(31/32): python3-markupsafe-0.23-19.el8.x86_64. 1.2 MB/s | 39 kB 00:00 2021-09-06T12:35:22.346 INFO:teuthology.orchestra.run.smithi041.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 392 kB/s | 85 kB 00:00 2021-09-06T12:35:22.349 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:35:22.349 INFO:teuthology.orchestra.run.smithi041.stdout:Total 11 MB/s | 28 MB 00:02 2021-09-06T12:35:22.358 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:35:22.399 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:35:22.399 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:35:22.630 INFO:teuthology.orchestra.run.smithi052.stdout:(2/32): ceph-mon-17.0.0-7472.g0eb1a794.el8.x86_ 8.2 MB/s | 3.9 MB 00:00 2021-09-06T12:35:22.633 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:35:22.637 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:35:22.854 INFO:teuthology.orchestra.run.smithi052.stdout:(3/32): ceph-mgr-17.0.0-7472.g0eb1a794.el8.x86_ 1.8 MB/s | 1.5 MB 00:00 2021-09-06T12:35:22.996 INFO:teuthology.orchestra.run.smithi052.stdout:(4/32): ceph-mgr-modules-core-17.0.0-7472.g0eb1 1.5 MB/s | 214 kB 00:00 2021-09-06T12:35:23.044 INFO:teuthology.orchestra.run.smithi052.stdout:(5/32): ceph-mds-17.0.0-7472.g0eb1a794.el8.x86_ 2.2 MB/s | 2.3 MB 00:01 2021-09-06T12:35:23.183 INFO:teuthology.orchestra.run.smithi052.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 232 kB/s | 44 kB 00:00 2021-09-06T12:35:23.342 INFO:teuthology.orchestra.run.smithi052.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2021-09-06T12:35:23.511 INFO:teuthology.orchestra.run.smithi052.stdout:(8/32): ceph-osd-17.0.0-7472.g0eb1a794.el8.x86_ 19 MB/s | 17 MB 00:00 2021-09-06T12:35:23.512 INFO:teuthology.orchestra.run.smithi052.stdout:(9/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 396 kB/s | 185 kB 00:00 2021-09-06T12:35:23.558 INFO:teuthology.orchestra.run.smithi052.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 242 kB/s | 11 kB 00:00 2021-09-06T12:35:23.608 INFO:teuthology.orchestra.run.smithi052.stdout:(11/32): python3-logutils-0.3.5-11.el8.noarch.r 962 kB/s | 49 kB 00:00 2021-09-06T12:35:23.613 INFO:teuthology.orchestra.run.smithi052.stdout:(12/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2021-09-06T12:35:23.660 INFO:teuthology.orchestra.run.smithi052.stdout:(13/32): python3-jaraco-functools-2.0-4.el8.noa 119 kB/s | 18 kB 00:00 2021-09-06T12:35:23.714 INFO:teuthology.orchestra.run.smithi052.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 557 kB/s | 59 kB 00:00 2021-09-06T12:35:23.715 INFO:teuthology.orchestra.run.smithi052.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 286 kB/s | 16 kB 00:00 2021-09-06T12:35:23.768 INFO:teuthology.orchestra.run.smithi052.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 347 kB/s | 19 kB 00:00 2021-09-06T12:35:23.769 INFO:teuthology.orchestra.run.smithi052.stdout:(17/32): python3-singledispatch-3.4.0.3-18.el8. 442 kB/s | 24 kB 00:00 2021-09-06T12:35:23.829 INFO:teuthology.orchestra.run.smithi052.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 484 kB/s | 29 kB 00:00 2021-09-06T12:35:23.830 INFO:teuthology.orchestra.run.smithi052.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 454 kB/s | 27 kB 00:00 2021-09-06T12:35:23.845 INFO:teuthology.orchestra.run.smithi052.stdout:(20/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2021-09-06T12:35:23.905 INFO:teuthology.orchestra.run.smithi052.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.4 MB/s | 85 kB 00:00 2021-09-06T12:35:23.963 INFO:teuthology.orchestra.run.smithi052.stdout:(22/32): python3-zc-lockfile-2.0-2.el8.noarch.r 396 kB/s | 23 kB 00:00 2021-09-06T12:35:24.004 INFO:teuthology.orchestra.run.smithi052.stdout:(23/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.8 MB/s | 157 kB 00:00 2021-09-06T12:35:24.030 INFO:teuthology.orchestra.run.smithi052.stdout:(24/32): python3-pytz-2017.2-9.el8.noarch.rpm 2.1 MB/s | 54 kB 00:00 2021-09-06T12:35:24.051 INFO:teuthology.orchestra.run.smithi052.stdout:(25/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2021-09-06T12:35:24.063 INFO:teuthology.orchestra.run.smithi052.stdout:(26/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 3.1 MB/s | 103 kB 00:00 2021-09-06T12:35:24.097 INFO:teuthology.orchestra.run.smithi052.stdout:(27/32): python3-waitress-1.2.1-2.el8.1.noarch. 900 kB/s | 241 kB 00:00 2021-09-06T12:35:24.099 INFO:teuthology.orchestra.run.smithi052.stdout:(28/32): python3-werkzeug-0.12.2-4.el8.noarch.r 9.4 MB/s | 457 kB 00:00 2021-09-06T12:35:24.100 INFO:teuthology.orchestra.run.smithi052.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 1.0 MB/s | 39 kB 00:00 2021-09-06T12:35:24.140 INFO:teuthology.orchestra.run.smithi052.stdout:(30/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.5 MB/s | 238 kB 00:00 2021-09-06T12:35:24.155 INFO:teuthology.orchestra.run.smithi052.stdout:(31/32): python3-pycparser-2.14-14.el8.noarch.r 1.9 MB/s | 109 kB 00:00 2021-09-06T12:35:24.169 INFO:teuthology.orchestra.run.smithi052.stdout:(32/32): python3-cryptography-3.2.1-4.el8.x86_6 10 MB/s | 559 kB 00:00 2021-09-06T12:35:24.171 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:35:24.171 INFO:teuthology.orchestra.run.smithi052.stdout:Total 13 MB/s | 28 MB 00:02 2021-09-06T12:35:24.180 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:35:24.221 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:35:24.222 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:35:24.437 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:35:24.442 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:35:25.213 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:35:25.429 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-06T12:35:25.632 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:35:25.638 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-06T12:35:25.885 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2021-09-06T12:35:25.894 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 3/32 2021-09-06T12:35:26.104 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2021-09-06T12:35:26.157 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 4/32 2021-09-06T12:35:26.349 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 5/32 2021-09-06T12:35:26.473 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 3/32 2021-09-06T12:35:26.558 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2021-09-06T12:35:26.765 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2021-09-06T12:35:26.796 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 4/32 2021-09-06T12:35:27.060 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 5/32 2021-09-06T12:35:27.113 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2021-09-06T12:35:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 6/32 2021-09-06T12:35:27.339 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 9/32 2021-09-06T12:35:27.358 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 7/32 2021-09-06T12:35:27.579 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 8/32 2021-09-06T12:35:27.617 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2021-09-06T12:35:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2021-09-06T12:35:27.816 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 9/32 2021-09-06T12:35:27.985 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2021-09-06T12:35:28.237 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 10/32 2021-09-06T12:35:28.250 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-06T12:35:28.458 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-06T12:35:28.584 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2021-09-06T12:35:28.639 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-06T12:35:28.766 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 12/32 2021-09-06T12:35:28.825 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-06T12:35:28.970 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-06T12:35:28.989 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2021-09-06T12:35:29.132 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-06T12:35:29.155 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 14/32 2021-09-06T12:35:29.350 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-06T12:35:29.369 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2021-09-06T12:35:29.531 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-06T12:35:29.555 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2021-09-06T12:35:29.732 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-06T12:35:29.733 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2021-09-06T12:35:29.870 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2021-09-06T12:35:29.991 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-06T12:35:30.114 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2021-09-06T12:35:30.210 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-06T12:35:30.303 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2021-09-06T12:35:30.436 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-06T12:35:30.496 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2021-09-06T12:35:30.776 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2021-09-06T12:35:30.780 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-06T12:35:31.016 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2021-09-06T12:35:31.114 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-06T12:35:31.253 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2021-09-06T12:35:31.558 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8. 27/32 2021-09-06T12:35:31.583 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2021-09-06T12:35:31.696 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 28/32 2021-09-06T12:35:31.952 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2021-09-06T12:35:32.326 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8. 27/32 2021-09-06T12:35:32.490 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 28/32 2021-09-06T12:35:33.400 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 28/32 2021-09-06T12:35:33.632 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 29/32 2021-09-06T12:35:34.049 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 29/32 2021-09-06T12:35:34.189 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 28/32 2021-09-06T12:35:34.216 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 30/32 2021-09-06T12:35:34.404 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 29/32 2021-09-06T12:35:34.446 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 30/32 2021-09-06T12:35:34.582 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 31/32 2021-09-06T12:35:34.601 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 31/32 2021-09-06T12:35:34.747 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 32/32 2021-09-06T12:35:34.820 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 29/32 2021-09-06T12:35:34.971 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 30/32 2021-09-06T12:35:35.112 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 32/32 2021-09-06T12:35:35.112 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/32 2021-09-06T12:35:35.113 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/32 2021-09-06T12:35:35.113 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/32 2021-09-06T12:35:35.113 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 4/32 2021-09-06T12:35:35.113 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/32 2021-09-06T12:35:35.113 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8. 6/32 2021-09-06T12:35:35.114 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-06T12:35:35.114 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-06T12:35:35.114 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-06T12:35:35.114 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-06T12:35:35.115 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-06T12:35:35.115 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-06T12:35:35.115 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-06T12:35:35.115 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-06T12:35:35.115 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-06T12:35:35.116 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-06T12:35:35.116 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-06T12:35:35.116 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-06T12:35:35.116 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-06T12:35:35.117 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-06T12:35:35.117 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-06T12:35:35.117 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-06T12:35:35.117 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-06T12:35:35.117 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-06T12:35:35.118 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 25/32 2021-09-06T12:35:35.118 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 26/32 2021-09-06T12:35:35.118 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 27/32 2021-09-06T12:35:35.118 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 28/32 2021-09-06T12:35:35.119 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 29/32 2021-09-06T12:35:35.119 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 30/32 2021-09-06T12:35:35.119 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2021-09-06T12:35:35.200 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 30/32 2021-09-06T12:35:35.345 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 31/32 2021-09-06T12:35:35.364 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 31/32 2021-09-06T12:35:35.461 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 32/32 2021-09-06T12:35:35.806 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 32/32 2021-09-06T12:35:35.807 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:35:35.828 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 32/32 2021-09-06T12:35:35.828 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/32 2021-09-06T12:35:35.828 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/32 2021-09-06T12:35:35.829 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 3/32 2021-09-06T12:35:35.829 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 4/32 2021-09-06T12:35:35.829 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 5/32 2021-09-06T12:35:35.829 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8. 6/32 2021-09-06T12:35:35.830 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2021-09-06T12:35:35.830 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2021-09-06T12:35:35.830 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2021-09-06T12:35:35.830 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2021-09-06T12:35:35.831 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2021-09-06T12:35:35.831 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2021-09-06T12:35:35.831 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2021-09-06T12:35:35.831 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2021-09-06T12:35:35.832 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2021-09-06T12:35:35.832 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2021-09-06T12:35:35.832 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2021-09-06T12:35:35.832 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2021-09-06T12:35:35.832 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2021-09-06T12:35:35.833 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2021-09-06T12:35:35.833 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 21/32 2021-09-06T12:35:35.833 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2021-09-06T12:35:35.833 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2021-09-06T12:35:35.834 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2021-09-06T12:35:35.834 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 25/32 2021-09-06T12:35:35.834 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 26/32 2021-09-06T12:35:35.834 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 27/32 2021-09-06T12:35:35.834 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 28/32 2021-09-06T12:35:35.835 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2021-09-06T12:35:35.835 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 30/32 2021-09-06T12:35:35.835 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 31/32 2021-09-06T12:35:36.616 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 32/32 2021-09-06T12:35:36.617 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:35:46.591 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:57 PM UTC. 2021-09-06T12:35:47.536 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:34:59 PM UTC. 2021-09-06T12:35:49.141 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:49.237 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:35:49.238 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:49.238 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:49.238 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:49.238 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:35:49.239 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:49.239 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:49.239 INFO:teuthology.orchestra.run.smithi041.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-06T12:35:49.239 INFO:teuthology.orchestra.run.smithi041.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-06T12:35:49.239 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-06T12:35:49.240 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-06T12:35:49.240 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-06T12:35:49.240 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-06T12:35:49.240 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-06T12:35:49.241 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-06T12:35:49.241 INFO:teuthology.orchestra.run.smithi041.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-06T12:35:49.241 INFO:teuthology.orchestra.run.smithi041.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-06T12:35:49.241 INFO:teuthology.orchestra.run.smithi041.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-06T12:35:49.241 INFO:teuthology.orchestra.run.smithi041.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-06T12:35:49.242 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-06T12:35:49.242 INFO:teuthology.orchestra.run.smithi041.stdout: python3-portend-2.6-1.el8.noarch 2021-09-06T12:35:49.242 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-06T12:35:49.242 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-06T12:35:49.243 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-06T12:35:49.243 INFO:teuthology.orchestra.run.smithi041.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-06T12:35:49.243 INFO:teuthology.orchestra.run.smithi041.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-06T12:35:49.243 INFO:teuthology.orchestra.run.smithi041.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-06T12:35:49.243 INFO:teuthology.orchestra.run.smithi041.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-06T12:35:49.244 INFO:teuthology.orchestra.run.smithi041.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-06T12:35:49.244 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-06T12:35:49.244 INFO:teuthology.orchestra.run.smithi041.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-06T12:35:49.244 INFO:teuthology.orchestra.run.smithi041.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-06T12:35:49.244 INFO:teuthology.orchestra.run.smithi041.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-06T12:35:49.245 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:35:49.245 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:35:49.299 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-base 2021-09-06T12:35:49.653 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:35:50.115 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:50.116 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:35:50.116 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:50.116 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:50.116 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:50.117 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:35:50.117 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:50.117 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:35:50.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-09-06T12:35:50.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-09-06T12:35:50.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-09-06T12:35:50.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-09-06T12:35:50.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-09-06T12:35:50.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-09-06T12:35:50.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-6.2-6.el8.noarch 2021-09-06T12:35:50.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-09-06T12:35:50.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-09-06T12:35:50.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.0.6-13.el8.noarch 2021-09-06T12:35:50.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-09-06T12:35:50.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-09-06T12:35:50.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-09-06T12:35:50.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-2.6-1.el8.noarch 2021-09-06T12:35:50.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-09-06T12:35:50.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.14-14.el8.noarch 2021-09-06T12:35:50.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2017.2-9.el8.noarch 2021-09-06T12:35:50.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-09-06T12:35:50.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-09-06T12:35:50.122 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-09-06T12:35:50.122 INFO:teuthology.orchestra.run.smithi052.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-09-06T12:35:50.122 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-09-06T12:35:50.122 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-09-06T12:35:50.122 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-09-06T12:35:50.123 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-09-06T12:35:50.123 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-09-06T12:35:50.123 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:35:50.123 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:35:50.216 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:35:50.235 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2021-09-06T12:35:50.586 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:35:51.123 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-06T12:35:55.556 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:35:56.323 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 kB/s | 2.4 kB 00:00 2021-09-06T12:36:00.865 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:36:01.678 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:36:03.110 INFO:teuthology.orchestra.run.smithi041.stdout:Package ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:36:03.559 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:36:03.560 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:36:03.561 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:36:03.682 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install cephadm 2021-09-06T12:36:04.009 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:36:04.028 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:04.449 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:36:04.450 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:36:04.451 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:36:04.599 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:36:04.739 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-06T12:36:04.879 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2021-09-06T12:36:05.228 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:05.758 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:36:05.946 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:36:06.661 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:36:06.661 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:06.661 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:36:06.662 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:06.662 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:36:06.662 INFO:teuthology.orchestra.run.smithi041.stdout: cephadm noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 69 k 2021-09-06T12:36:06.662 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:06.662 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:36:06.663 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:06.663 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:36:06.663 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:06.663 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 69 k 2021-09-06T12:36:06.664 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 309 k 2021-09-06T12:36:06.664 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:36:06.792 INFO:teuthology.orchestra.run.smithi041.stdout:cephadm-17.0.0-7472.g0eb1a794.el8.noarch.rpm 533 kB/s | 69 kB 00:00 2021-09-06T12:36:06.793 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:36:06.793 INFO:teuthology.orchestra.run.smithi041.stdout:Total 527 kB/s | 69 kB 00:00 2021-09-06T12:36:06.794 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:36:06.801 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:36:06.801 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:36:06.803 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:36:06.807 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:36:06.852 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:36:07.890 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:36:07.891 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:07.891 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:36:07.892 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:07.892 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:36:07.892 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 69 k 2021-09-06T12:36:07.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:07.892 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:36:07.893 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:07.893 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:36:07.893 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:07.893 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 69 k 2021-09-06T12:36:07.893 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 309 k 2021-09-06T12:36:07.894 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:36:08.056 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-17.0.0-7472.g0eb1a794.el8.noarch.rpm 423 kB/s | 69 kB 00:00 2021-09-06T12:36:08.057 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:36:08.057 INFO:teuthology.orchestra.run.smithi052.stdout:Total 420 kB/s | 69 kB 00:00 2021-09-06T12:36:08.058 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:36:08.065 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:36:08.065 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:36:08.067 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:36:08.071 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:36:08.121 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:36:09.404 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:09.554 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:09.723 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:09.953 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:09.953 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:36:10.807 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:11.055 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:11.225 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:11.523 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/1 2021-09-06T12:36:11.524 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:36:20.771 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:35:41 PM UTC. 2021-09-06T12:36:22.179 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:35:42 PM UTC. 2021-09-06T12:36:23.338 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:23.338 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:36:23.338 INFO:teuthology.orchestra.run.smithi041.stdout: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:36:23.338 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:23.339 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:36:23.466 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-immutable-object-cache 2021-09-06T12:36:23.826 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:24.431 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:36:24.798 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:24.799 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:36:24.799 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:36:24.799 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:24.799 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:36:24.925 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2021-09-06T12:36:25.290 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:25.852 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-06T12:36:29.666 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-06T12:36:30.955 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:36:34.913 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:36:36.331 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:36:37.587 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:36:37.588 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:37.588 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repo Size 2021-09-06T12:36:37.588 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:37.589 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:36:37.589 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 169 k 2021-09-06T12:36:37.589 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:37.589 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:36:37.590 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:36:37.590 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:36:37.590 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:37.590 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 169 k 2021-09-06T12:36:37.590 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 567 k 2021-09-06T12:36:37.591 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:36:38.548 INFO:teuthology.orchestra.run.smithi041.stdout:ceph-immutable-object-cache-17.0.0-7472.g0eb1a7 176 kB/s | 169 kB 00:00 2021-09-06T12:36:38.549 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:36:38.549 INFO:teuthology.orchestra.run.smithi041.stdout:Total 176 kB/s | 169 kB 00:00 2021-09-06T12:36:38.550 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:36:38.566 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:36:38.566 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:36:38.628 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:36:38.633 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:36:38.759 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:36:38.940 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:39.137 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:36:39.138 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:39.139 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2021-09-06T12:36:39.139 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:39.140 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:36:39.140 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 169 k 2021-09-06T12:36:39.140 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:39.141 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:36:39.141 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:36:39.142 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:36:39.142 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:39.143 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 169 k 2021-09-06T12:36:39.143 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 567 k 2021-09-06T12:36:39.143 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:36:39.299 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:39.413 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-17.0.0-7472.g0eb1a7 617 kB/s | 169 kB 00:00 2021-09-06T12:36:39.413 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:36:39.414 INFO:teuthology.orchestra.run.smithi052.stdout:Total 613 kB/s | 169 kB 00:00 2021-09-06T12:36:39.414 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:36:39.428 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:36:39.429 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:36:39.486 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:36:39.490 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:36:39.549 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:39.550 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:36:39.609 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:36:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:40.128 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:40.392 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794. 1/1 2021-09-06T12:36:40.393 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:36:50.336 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:36:15 PM UTC. 2021-09-06T12:36:51.039 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:36:16 PM UTC. 2021-09-06T12:36:52.893 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:52.894 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:36:52.894 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:36:52.894 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:36:52.894 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:36:53.042 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-mgr 2021-09-06T12:36:53.393 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:53.626 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:53.626 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:36:53.626 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:36:53.626 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:36:53.627 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:36:53.756 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2021-09-06T12:36:53.978 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:36:54.107 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:36:54.642 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-06T12:36:59.209 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:36:59.734 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2021-09-06T12:37:04.404 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:37:05.341 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:37:06.668 INFO:teuthology.orchestra.run.smithi041.stdout:Package ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:37:07.123 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:37:07.125 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:37:07.125 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:37:07.230 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-mgr-dashboard 2021-09-06T12:37:07.568 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:37:07.578 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:37:08.010 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:37:08.011 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:37:08.011 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:37:08.119 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2021-09-06T12:37:08.135 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:37:08.270 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-06T12:37:08.465 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:37:09.024 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-06T12:37:09.211 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:37:10.180 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:37:10.181 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================== 2021-09-06T12:37:10.181 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:37:10.182 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================== 2021-09-06T12:37:10.182 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:37:10.182 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 1.5 M 2021-09-06T12:37:10.182 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:37:10.183 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 24 k 2021-09-06T12:37:10.183 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 8.9 k 2021-09-06T12:37:10.183 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-06T12:37:10.183 INFO:teuthology.orchestra.run.smithi041.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-06T12:37:10.184 INFO:teuthology.orchestra.run.smithi041.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-06T12:37:10.184 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:10.184 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:37:10.184 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================== 2021-09-06T12:37:10.184 INFO:teuthology.orchestra.run.smithi041.stdout:Install 6 Packages 2021-09-06T12:37:10.185 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:10.185 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 1.8 M 2021-09-06T12:37:10.186 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 7.4 M 2021-09-06T12:37:10.186 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:37:10.285 INFO:teuthology.orchestra.run.smithi041.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7472.g0eb1 88 kB/s | 8.9 kB 00:00 2021-09-06T12:37:10.295 INFO:teuthology.orchestra.run.smithi041.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7472.g0eb 215 kB/s | 24 kB 00:00 2021-09-06T12:37:10.367 INFO:teuthology.orchestra.run.smithi041.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 409 kB/s | 34 kB 00:00 2021-09-06T12:37:10.408 INFO:teuthology.orchestra.run.smithi041.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2021-09-06T12:37:10.430 INFO:teuthology.orchestra.run.smithi041.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-09-06T12:37:10.507 INFO:teuthology.orchestra.run.smithi041.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7472.g0eb1a794 4.7 MB/s | 1.5 MB 00:00 2021-09-06T12:37:10.509 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:37:10.510 INFO:teuthology.orchestra.run.smithi041.stdout:Total 5.6 MB/s | 1.8 MB 00:00 2021-09-06T12:37:10.511 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:37:10.520 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:37:10.520 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:37:10.577 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:37:10.581 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:37:10.847 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:37:11.030 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-06T12:37:11.186 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:37:11.187 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================== 2021-09-06T12:37:11.187 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:37:11.187 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================== 2021-09-06T12:37:11.188 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:37:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 1.5 M 2021-09-06T12:37:11.188 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:37:11.188 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 24 k 2021-09-06T12:37:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 8.9 k 2021-09-06T12:37:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-06T12:37:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-09-06T12:37:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-09-06T12:37:11.189 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:11.190 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:37:11.190 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================== 2021-09-06T12:37:11.190 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2021-09-06T12:37:11.190 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:11.191 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.8 M 2021-09-06T12:37:11.191 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 7.4 M 2021-09-06T12:37:11.192 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:37:11.248 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-06T12:37:11.297 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7472.g0eb1 83 kB/s | 8.9 kB 00:00 2021-09-06T12:37:11.300 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7472.g0eb 216 kB/s | 24 kB 00:00 2021-09-06T12:37:11.380 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 409 kB/s | 34 kB 00:00 2021-09-06T12:37:11.419 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.6 MB/s | 196 kB 00:00 2021-09-06T12:37:11.422 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2021-09-06T12:37:11.511 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7472.g0eb1a794 4.8 MB/s | 1.5 MB 00:00 2021-09-06T12:37:11.513 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:37:11.514 INFO:teuthology.orchestra.run.smithi052.stdout:Total 5.6 MB/s | 1.8 MB 00:00 2021-09-06T12:37:11.515 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:37:11.523 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:37:11.524 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:37:11.563 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-06T12:37:11.580 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:37:11.584 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:37:11.669 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.n 4/6 2021-09-06T12:37:11.867 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:37:11.998 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8. 5/6 2021-09-06T12:37:12.061 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2021-09-06T12:37:12.207 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 6/6 2021-09-06T12:37:12.236 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2021-09-06T12:37:12.439 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 6/6 2021-09-06T12:37:12.439 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8. 1/6 2021-09-06T12:37:12.439 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 2/6 2021-09-06T12:37:12.440 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.n 3/6 2021-09-06T12:37:12.440 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-06T12:37:12.440 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-06T12:37:12.443 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2021-09-06T12:37:12.591 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.n 4/6 2021-09-06T12:37:12.845 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-06T12:37:12.845 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:37:12.963 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8. 5/6 2021-09-06T12:37:13.176 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 6/6 2021-09-06T12:37:13.428 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 6/6 2021-09-06T12:37:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8. 1/6 2021-09-06T12:37:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarc 2/6 2021-09-06T12:37:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.n 3/6 2021-09-06T12:37:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2021-09-06T12:37:13.429 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2021-09-06T12:37:13.729 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2021-09-06T12:37:13.729 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:37:23.535 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:36:45 PM UTC. 2021-09-06T12:37:24.543 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:36:45 PM UTC. 2021-09-06T12:37:26.146 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:26.147 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:37:26.147 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:26.147 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:26.147 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:26.148 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-06T12:37:26.148 INFO:teuthology.orchestra.run.smithi041.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-06T12:37:26.148 INFO:teuthology.orchestra.run.smithi041.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-06T12:37:26.148 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:26.148 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:37:26.267 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-06T12:37:26.616 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:37:27.215 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:27.215 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:37:27.215 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:27.215 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:27.216 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:27.216 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-09-06T12:37:27.216 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-09-06T12:37:27.216 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.4.1-12.el8.noarch 2021-09-06T12:37:27.216 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:27.217 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:37:27.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2021-09-06T12:37:27.367 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-06T12:37:27.681 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:37:28.215 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-06T12:37:32.697 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 70 kB/s | 2.8 kB 00:00 2021-09-06T12:37:33.309 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2021-09-06T12:37:37.846 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:37:38.704 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:37:40.571 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:37:40.573 INFO:teuthology.orchestra.run.smithi041.stdout:====================================================================================================================== 2021-09-06T12:37:40.573 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:37:40.573 INFO:teuthology.orchestra.run.smithi041.stdout:====================================================================================================================== 2021-09-06T12:37:40.573 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:37:40.574 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 7.4 M 2021-09-06T12:37:40.574 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:37:40.574 INFO:teuthology.orchestra.run.smithi041.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-06T12:37:40.574 INFO:teuthology.orchestra.run.smithi041.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-06T12:37:40.575 INFO:teuthology.orchestra.run.smithi041.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-06T12:37:40.575 INFO:teuthology.orchestra.run.smithi041.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-06T12:37:40.575 INFO:teuthology.orchestra.run.smithi041.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-06T12:37:40.575 INFO:teuthology.orchestra.run.smithi041.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-06T12:37:40.576 INFO:teuthology.orchestra.run.smithi041.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-06T12:37:40.576 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:40.576 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:37:40.576 INFO:teuthology.orchestra.run.smithi041.stdout:====================================================================================================================== 2021-09-06T12:37:40.577 INFO:teuthology.orchestra.run.smithi041.stdout:Install 8 Packages 2021-09-06T12:37:40.577 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:40.578 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 36 M 2021-09-06T12:37:40.578 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 199 M 2021-09-06T12:37:40.578 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:37:40.628 INFO:teuthology.orchestra.run.smithi041.stdout:(1/8): libquadmath-8.4.1-1.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2021-09-06T12:37:40.655 INFO:teuthology.orchestra.run.smithi041.stdout:(2/8): libgfortran-8.4.1-1.el8.x86_64.rpm 7.9 MB/s | 639 kB 00:00 2021-09-06T12:37:40.704 INFO:teuthology.orchestra.run.smithi041.stdout:(3/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2021-09-06T12:37:40.774 INFO:teuthology.orchestra.run.smithi041.stdout:(4/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 68 MB/s | 4.7 MB 00:00 2021-09-06T12:37:40.924 INFO:teuthology.orchestra.run.smithi041.stdout:(5/8): python3-scipy-1.0.0-20.module+el8.1.0+33 54 MB/s | 15 MB 00:00 2021-09-06T12:37:40.959 INFO:teuthology.orchestra.run.smithi041.stdout:(6/8): openblas-0.3.12-1.el8.x86_64.rpm 25 MB/s | 4.6 MB 00:00 2021-09-06T12:37:40.960 INFO:teuthology.orchestra.run.smithi041.stdout:(7/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 6.2 MB/s | 225 kB 00:00 2021-09-06T12:37:41.423 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:37:41.424 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================================================== 2021-09-06T12:37:41.424 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:37:41.424 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================================================== 2021-09-06T12:37:41.425 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:37:41.425 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 7.4 M 2021-09-06T12:37:41.425 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:37:41.425 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2021-09-06T12:37:41.425 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2021-09-06T12:37:41.426 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2021-09-06T12:37:41.426 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2021-09-06T12:37:41.426 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2021-09-06T12:37:41.426 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2021-09-06T12:37:41.426 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2021-09-06T12:37:41.427 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:41.427 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:37:41.427 INFO:teuthology.orchestra.run.smithi052.stdout:====================================================================================================================== 2021-09-06T12:37:41.427 INFO:teuthology.orchestra.run.smithi052.stdout:Install 8 Packages 2021-09-06T12:37:41.427 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:37:41.428 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 36 M 2021-09-06T12:37:41.428 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 199 M 2021-09-06T12:37:41.429 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:37:41.509 INFO:teuthology.orchestra.run.smithi052.stdout:(1/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2021-09-06T12:37:41.574 INFO:teuthology.orchestra.run.smithi052.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 74 MB/s | 4.7 MB 00:00 2021-09-06T12:37:41.639 INFO:teuthology.orchestra.run.smithi052.stdout:(3/8): openblas-0.3.12-1.el8.x86_64.rpm 71 MB/s | 4.6 MB 00:00 2021-09-06T12:37:41.748 INFO:teuthology.orchestra.run.smithi052.stdout:(4/8): python3-scipy-1.0.0-20.module+el8.1.0+33 45 MB/s | 15 MB 00:00 2021-09-06T12:37:41.780 INFO:teuthology.orchestra.run.smithi052.stdout:(5/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 1.6 MB/s | 225 kB 00:00 2021-09-06T12:37:41.792 INFO:teuthology.orchestra.run.smithi052.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 4.4 MB/s | 169 kB 00:00 2021-09-06T12:37:41.811 INFO:teuthology.orchestra.run.smithi052.stdout:(7/8): libgfortran-8.4.1-1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2021-09-06T12:37:42.057 INFO:teuthology.orchestra.run.smithi041.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-747 5.0 MB/s | 7.4 MB 00:01 2021-09-06T12:37:42.059 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:37:42.059 INFO:teuthology.orchestra.run.smithi041.stdout:Total 24 MB/s | 36 MB 00:01 2021-09-06T12:37:42.069 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:37:42.090 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:37:42.090 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:37:42.407 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:37:42.412 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:37:42.542 INFO:teuthology.orchestra.run.smithi052.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-747 6.6 MB/s | 7.4 MB 00:01 2021-09-06T12:37:42.543 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:37:42.544 INFO:teuthology.orchestra.run.smithi052.stdout:Total 32 MB/s | 36 MB 00:01 2021-09-06T12:37:42.554 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:37:42.575 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:37:42.575 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:37:42.893 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:37:42.898 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:37:42.991 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:37:43.148 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T12:37:43.287 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T12:37:43.489 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T12:37:43.672 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:37:43.854 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T12:37:43.978 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-09-06T12:37:44.064 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T12:37:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T12:37:44.289 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T12:37:44.753 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-09-06T12:37:44.853 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T12:37:44.928 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T12:37:45.006 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-06T12:37:45.521 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-09-06T12:37:45.679 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.12-1.el8.x86_64 4/8 2021-09-06T12:37:45.794 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-06T12:37:46.263 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T12:37:46.459 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 4/8 2021-09-06T12:37:46.472 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T12:37:46.773 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-09-06T12:37:46.978 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T12:37:48.639 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T12:37:49.101 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T12:37:49.932 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-06T12:37:50.063 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 8/8 2021-09-06T12:37:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 8/8 2021-09-06T12:37:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 1/8 2021-09-06T12:37:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 2/8 2021-09-06T12:37:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 3/8 2021-09-06T12:37:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-09-06T12:37:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 5/8 2021-09-06T12:37:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 6/8 2021-09-06T12:37:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 7/8 2021-09-06T12:37:50.405 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 7/8 2021-09-06T12:37:50.595 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 8/8 2021-09-06T12:37:50.627 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 8/8 2021-09-06T12:37:50.628 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:37:50.852 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 8/8 2021-09-06T12:37:50.853 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a79 1/8 2021-09-06T12:37:50.853 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 2/8 2021-09-06T12:37:50.853 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7. 3/8 2021-09-06T12:37:50.853 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-09-06T12:37:50.853 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 5/8 2021-09-06T12:37:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-09-06T12:37:50.854 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 7/8 2021-09-06T12:37:51.145 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 8/8 2021-09-06T12:37:51.145 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:37:56.161 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:56.162 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:37:56.162 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:37:56.162 INFO:teuthology.orchestra.run.smithi041.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-06T12:37:56.163 INFO:teuthology.orchestra.run.smithi041.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-06T12:37:56.163 INFO:teuthology.orchestra.run.smithi041.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-06T12:37:56.163 INFO:teuthology.orchestra.run.smithi041.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-06T12:37:56.163 INFO:teuthology.orchestra.run.smithi041.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-06T12:37:56.164 INFO:teuthology.orchestra.run.smithi041.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-06T12:37:56.164 INFO:teuthology.orchestra.run.smithi041.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-06T12:37:56.164 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:37:56.164 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:37:56.294 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-mgr-rook 2021-09-06T12:37:56.647 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:37:57.280 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:38:02.030 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:37:19 PM UTC. 2021-09-06T12:38:02.510 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:38:04.659 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:04.660 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:38:04.660 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:38:04.660 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-09-06T12:38:04.660 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-09-06T12:38:04.661 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.12-1.el8.x86_64 2021-09-06T12:38:04.661 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-09-06T12:38:04.661 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-09-06T12:38:04.661 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-09-06T12:38:04.662 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2021-09-06T12:38:04.662 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:04.662 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:38:04.789 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2021-09-06T12:38:05.139 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:38:05.662 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 71 kB/s | 2.8 kB 00:00 2021-09-06T12:38:07.670 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:38:10.374 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:38:10.376 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================== 2021-09-06T12:38:10.376 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:38:10.376 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================== 2021-09-06T12:38:10.376 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:38:10.377 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-rook noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 44 k 2021-09-06T12:38:10.377 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:38:10.377 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-06T12:38:10.377 INFO:teuthology.orchestra.run.smithi041.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-06T12:38:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-06T12:38:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-06T12:38:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-06T12:38:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-06T12:38:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-06T12:38:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-06T12:38:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-06T12:38:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-06T12:38:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-06T12:38:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-06T12:38:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:10.380 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:38:10.380 INFO:teuthology.orchestra.run.smithi041.stdout:======================================================================================================== 2021-09-06T12:38:10.380 INFO:teuthology.orchestra.run.smithi041.stdout:Install 13 Packages 2021-09-06T12:38:10.381 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:10.381 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 1.9 M 2021-09-06T12:38:10.382 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 25 M 2021-09-06T12:38:10.382 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:38:10.469 INFO:teuthology.orchestra.run.smithi041.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 366 kB/s | 33 kB 00:00 2021-09-06T12:38:10.470 INFO:teuthology.orchestra.run.smithi041.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 181 kB/s | 16 kB 00:00 2021-09-06T12:38:10.526 INFO:teuthology.orchestra.run.smithi041.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 1.4 MB/s | 81 kB 00:00 2021-09-06T12:38:10.529 INFO:teuthology.orchestra.run.smithi041.stdout:(4/13): ceph-mgr-rook-17.0.0-7472.g0eb1a794.el8 291 kB/s | 44 kB 00:00 2021-09-06T12:38:10.555 INFO:teuthology.orchestra.run.smithi041.stdout:(5/13): python3-rsa-3.4.2-16.el8.noarch.rpm 2.5 MB/s | 73 kB 00:00 2021-09-06T12:38:10.596 INFO:teuthology.orchestra.run.smithi041.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 1.0 MB/s | 43 kB 00:00 2021-09-06T12:38:10.624 INFO:teuthology.orchestra.run.smithi041.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 5.4 MB/s | 155 kB 00:00 2021-09-06T12:38:10.645 INFO:teuthology.orchestra.run.smithi041.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 532 kB/s | 61 kB 00:00 2021-09-06T12:38:10.656 INFO:teuthology.orchestra.run.smithi041.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 841 kB/s | 27 kB 00:00 2021-09-06T12:38:10.676 INFO:teuthology.orchestra.run.smithi041.stdout:(10/13): python3-kubernetes-11.0.0-6.el8.noarch 5.7 MB/s | 1.2 MB 00:00 2021-09-06T12:38:10.690 INFO:teuthology.orchestra.run.smithi041.stdout:(11/13): python3-jsonpointer-1.10-11.el8.noarch 586 kB/s | 20 kB 00:00 2021-09-06T12:38:10.700 INFO:teuthology.orchestra.run.smithi041.stdout:(12/13): python3-pyasn1-modules-0.3.7-6.el8.noa 1.9 MB/s | 110 kB 00:00 2021-09-06T12:38:10.727 INFO:teuthology.orchestra.run.smithi041.stdout:(13/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.8 MB/s | 126 kB 00:00 2021-09-06T12:38:10.727 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:38:10.728 INFO:teuthology.orchestra.run.smithi041.stdout:Total 5.5 MB/s | 1.9 MB 00:00 2021-09-06T12:38:10.734 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:38:10.749 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:38:10.749 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:38:10.804 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:38:10.824 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:38:10.828 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:38:11.324 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:38:11.590 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-06T12:38:11.787 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-06T12:38:12.001 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-06T12:38:12.203 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2021-09-06T12:38:12.415 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2021-09-06T12:38:12.662 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-09-06T12:38:12.902 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-09-06T12:38:13.062 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-06T12:38:13.257 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-06T12:38:13.506 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-06T12:38:14.695 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-06T12:38:15.072 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-06T12:38:15.952 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 13/13 2021-09-06T12:38:16.204 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 13/13 2021-09-06T12:38:16.205 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 1/13 2021-09-06T12:38:16.205 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-06T12:38:16.205 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-06T12:38:16.205 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-06T12:38:16.206 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-06T12:38:16.206 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-06T12:38:16.206 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-06T12:38:16.206 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2021-09-06T12:38:16.207 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2021-09-06T12:38:16.207 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 10/13 2021-09-06T12:38:16.207 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 11/13 2021-09-06T12:38:16.207 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 12/13 2021-09-06T12:38:16.227 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:38:16.540 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 13/13 2021-09-06T12:38:16.540 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:38:19.114 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:38:19.116 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================== 2021-09-06T12:38:19.116 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:38:19.117 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================== 2021-09-06T12:38:19.117 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:38:19.117 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 44 k 2021-09-06T12:38:19.117 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:38:19.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-09-06T12:38:19.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-09-06T12:38:19.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-09-06T12:38:19.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2021-09-06T12:38:19.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2021-09-06T12:38:19.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-09-06T12:38:19.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2021-09-06T12:38:19.119 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2021-09-06T12:38:19.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2021-09-06T12:38:19.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2021-09-06T12:38:19.120 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-09-06T12:38:19.121 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-09-06T12:38:19.121 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:19.121 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:38:19.121 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================================== 2021-09-06T12:38:19.122 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2021-09-06T12:38:19.122 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:19.122 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 1.9 M 2021-09-06T12:38:19.123 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 25 M 2021-09-06T12:38:19.123 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:38:19.200 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 410 kB/s | 33 kB 00:00 2021-09-06T12:38:19.247 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 127 kB/s | 16 kB 00:00 2021-09-06T12:38:19.317 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 693 kB/s | 81 kB 00:00 2021-09-06T12:38:19.375 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): ceph-mgr-rook-17.0.0-7472.g0eb1a794.el8 172 kB/s | 44 kB 00:00 2021-09-06T12:38:19.395 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-rsa-3.4.2-16.el8.noarch.rpm 939 kB/s | 73 kB 00:00 2021-09-06T12:38:19.443 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 562 kB/s | 27 kB 00:00 2021-09-06T12:38:19.479 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-pyasn1-modules-0.3.7-6.el8.noar 3.0 MB/s | 110 kB 00:00 2021-09-06T12:38:19.491 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-websocket-client-0.56.0-5.el8.n 530 kB/s | 61 kB 00:00 2021-09-06T12:38:19.523 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-kubernetes-11.0.0-6.el8.noarch. 4.2 MB/s | 1.2 MB 00:00 2021-09-06T12:38:19.579 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-jsonpointer-1.10-11.el8.noarch 201 kB/s | 20 kB 00:00 2021-09-06T12:38:19.650 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 797 kB/s | 126 kB 00:00 2021-09-06T12:38:19.686 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 1.5 MB/s | 155 kB 00:00 2021-09-06T12:38:19.709 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-requests-oauthlib-1.0.0-1.el8. 247 kB/s | 43 kB 00:00 2021-09-06T12:38:19.710 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:38:19.711 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.3 MB/s | 1.9 MB 00:00 2021-09-06T12:38:19.717 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:38:19.733 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:38:19.734 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:38:19.826 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:38:19.831 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:38:21.209 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:38:21.488 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-09-06T12:38:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 2/13 2021-09-06T12:38:21.922 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2021-09-06T12:38:22.115 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 4/13 2021-09-06T12:38:22.298 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 5/13 2021-09-06T12:38:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 6/13 2021-09-06T12:38:22.707 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 7/13 2021-09-06T12:38:22.858 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2021-09-06T12:38:23.045 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-09-06T12:38:23.309 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2021-09-06T12:38:24.650 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2021-09-06T12:38:24.901 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2021-09-06T12:38:25.090 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 13/13 2021-09-06T12:38:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 13/13 2021-09-06T12:38:25.329 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 1/13 2021-09-06T12:38:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2021-09-06T12:38:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2021-09-06T12:38:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2021-09-06T12:38:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2021-09-06T12:38:25.330 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 6/13 2021-09-06T12:38:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2021-09-06T12:38:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 8/13 2021-09-06T12:38:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 9/13 2021-09-06T12:38:25.331 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 10/13 2021-09-06T12:38:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 11/13 2021-09-06T12:38:25.332 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 12/13 2021-09-06T12:38:25.694 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 13/13 2021-09-06T12:38:25.694 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:38:27.211 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:37:54 PM UTC. 2021-09-06T12:38:29.839 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:29.840 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:38:29.840 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:38:29.840 INFO:teuthology.orchestra.run.smithi041.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-06T12:38:29.840 INFO:teuthology.orchestra.run.smithi041.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-06T12:38:29.841 INFO:teuthology.orchestra.run.smithi041.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-06T12:38:29.841 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-06T12:38:29.841 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-06T12:38:29.841 INFO:teuthology.orchestra.run.smithi041.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-06T12:38:29.841 INFO:teuthology.orchestra.run.smithi041.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-06T12:38:29.842 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-06T12:38:29.842 INFO:teuthology.orchestra.run.smithi041.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-06T12:38:29.842 INFO:teuthology.orchestra.run.smithi041.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-06T12:38:29.842 INFO:teuthology.orchestra.run.smithi041.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-06T12:38:29.842 INFO:teuthology.orchestra.run.smithi041.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-06T12:38:29.843 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:29.843 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:38:29.953 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-mgr-cephadm 2021-09-06T12:38:30.311 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:38:30.889 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2021-09-06T12:38:36.129 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 69 kB/s | 2.8 kB 00:00 2021-09-06T12:38:36.470 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:37:56 PM UTC. 2021-09-06T12:38:39.115 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:39.115 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:38:39.115 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:38:39.116 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-09-06T12:38:39.116 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-09-06T12:38:39.116 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-09-06T12:38:39.116 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-09-06T12:38:39.116 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-09-06T12:38:39.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-09-06T12:38:39.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-09-06T12:38:39.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-09-06T12:38:39.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-09-06T12:38:39.117 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-09-06T12:38:39.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-09-06T12:38:39.118 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-09-06T12:38:39.118 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:39.118 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:38:39.234 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2021-09-06T12:38:39.591 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:38:40.126 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-06T12:38:41.294 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:38:44.022 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:38:44.023 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================================== 2021-09-06T12:38:44.023 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:38:44.023 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================================== 2021-09-06T12:38:44.023 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:38:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 91 k 2021-09-06T12:38:44.024 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:38:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-06T12:38:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-06T12:38:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-06T12:38:44.025 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:44.025 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:38:44.025 INFO:teuthology.orchestra.run.smithi041.stdout:=================================================================================================================== 2021-09-06T12:38:44.025 INFO:teuthology.orchestra.run.smithi041.stdout:Install 4 Packages 2021-09-06T12:38:44.026 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:38:44.026 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 5.9 M 2021-09-06T12:38:44.026 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 25 M 2021-09-06T12:38:44.027 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:38:44.065 INFO:teuthology.orchestra.run.smithi041.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 13 MB/s | 538 kB 00:00 2021-09-06T12:38:44.126 INFO:teuthology.orchestra.run.smithi041.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 79 MB/s | 4.8 MB 00:00 2021-09-06T12:38:44.196 INFO:teuthology.orchestra.run.smithi041.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7472.g0eb1a794.e 530 kB/s | 91 kB 00:00 2021-09-06T12:38:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:38:46.638 INFO:teuthology.orchestra.run.smithi041.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 182 kB/s | 476 kB 00:02 2021-09-06T12:38:46.640 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:38:46.641 INFO:teuthology.orchestra.run.smithi041.stdout:Total 2.2 MB/s | 5.9 MB 00:02 2021-09-06T12:38:46.643 INFO:teuthology.orchestra.run.smithi041.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-06T12:38:47.411 INFO:teuthology.orchestra.run.smithi041.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 1.0 kB 00:00 2021-09-06T12:38:47.502 INFO:teuthology.orchestra.run.smithi041.stderr:Importing GPG key 0x45BF1189: 2021-09-06T12:38:47.503 INFO:teuthology.orchestra.run.smithi041.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-06T12:38:47.503 INFO:teuthology.orchestra.run.smithi041.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-06T12:38:47.503 INFO:teuthology.orchestra.run.smithi041.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-06T12:38:47.722 INFO:teuthology.orchestra.run.smithi041.stdout:Key imported successfully 2021-09-06T12:38:47.727 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:38:47.738 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:38:47.738 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:38:47.824 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:38:47.829 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:38:49.237 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:38:49.746 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-06T12:38:50.120 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-06T12:38:50.381 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 3/4 2021-09-06T12:38:50.581 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 4/4 2021-09-06T12:38:50.603 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:38:50.803 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 4/4 2021-09-06T12:38:50.803 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/4 2021-09-06T12:38:50.803 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 2/4 2021-09-06T12:38:50.804 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T12:38:51.064 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-06T12:38:51.064 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:38:53.328 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:38:53.329 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================================== 2021-09-06T12:38:53.329 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:38:53.330 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================================== 2021-09-06T12:38:53.330 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:38:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7472.g0eb1a794.el8 ceph-noarch 91 k 2021-09-06T12:38:53.330 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:38:53.330 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.7.0-1.el8 copr:copr.fedorainfracloud.org:ceph:python3-asyncssh 476 k 2021-09-06T12:38:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2021-09-06T12:38:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2021-09-06T12:38:53.331 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:53.331 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:38:53.331 INFO:teuthology.orchestra.run.smithi052.stdout:=================================================================================================================== 2021-09-06T12:38:53.332 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2021-09-06T12:38:53.332 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:38:53.332 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 5.9 M 2021-09-06T12:38:53.333 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 25 M 2021-09-06T12:38:53.333 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:38:53.378 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2021-09-06T12:38:53.434 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): python3-babel-2.5.1-5.el8.noarch.rpm 85 MB/s | 4.8 MB 00:00 2021-09-06T12:38:53.486 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): ceph-mgr-cephadm-17.0.0-7472.g0eb1a794.e 586 kB/s | 91 kB 00:00 2021-09-06T12:38:55.193 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): python3-asyncssh-2.7.0-1.el8.noarch.rpm 256 kB/s | 476 kB 00:01 2021-09-06T12:38:55.196 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:38:55.196 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.1 MB/s | 5.9 MB 00:01 2021-09-06T12:38:55.198 INFO:teuthology.orchestra.run.smithi052.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ceph:python3-asyncssh-5dfbfa110d1e0dad/packages/python3-asyncssh-2.7.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 45bf1189: NOKEY 2021-09-06T12:38:55.966 INFO:teuthology.orchestra.run.smithi052.stdout:Copr repo for python3-asyncssh owned by ceph 1.3 kB/s | 1.0 kB 00:00 2021-09-06T12:38:56.061 INFO:teuthology.orchestra.run.smithi052.stderr:Importing GPG key 0x45BF1189: 2021-09-06T12:38:56.061 INFO:teuthology.orchestra.run.smithi052.stderr: Userid : "ceph_python3-asyncssh (None) " 2021-09-06T12:38:56.061 INFO:teuthology.orchestra.run.smithi052.stderr: Fingerprint: FD53 FC11 9FA9 2C47 E35D 9C35 339B 8950 45BF 1189 2021-09-06T12:38:56.062 INFO:teuthology.orchestra.run.smithi052.stderr: From : https://download.copr.fedorainfracloud.org/results/ceph/python3-asyncssh/pubkey.gpg 2021-09-06T12:38:56.186 INFO:teuthology.orchestra.run.smithi052.stdout:Key imported successfully 2021-09-06T12:38:56.191 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:38:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:38:56.202 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:38:56.289 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:38:56.293 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:38:57.328 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:38:57.707 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 1/4 2021-09-06T12:38:58.234 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 2/4 2021-09-06T12:38:58.530 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.7.0-1.el8.noarch 3/4 2021-09-06T12:38:58.694 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 4/4 2021-09-06T12:38:58.951 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 4/4 2021-09-06T12:38:58.951 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 1/4 2021-09-06T12:38:58.952 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.7.0-1.el8.noarch 2/4 2021-09-06T12:38:58.952 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/4 2021-09-06T12:38:59.244 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 4/4 2021-09-06T12:38:59.245 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:39:01.736 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:38:22 PM UTC. 2021-09-06T12:39:04.392 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:04.392 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:39:04.392 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:39:04.393 INFO:teuthology.orchestra.run.smithi041.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-06T12:39:04.393 INFO:teuthology.orchestra.run.smithi041.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-06T12:39:04.393 INFO:teuthology.orchestra.run.smithi041.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-06T12:39:04.393 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:04.393 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:39:04.514 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install ceph-fuse 2021-09-06T12:39:04.875 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:39:05.450 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 60 kB/s | 2.4 kB 00:00 2021-09-06T12:39:10.134 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:38:31 PM UTC. 2021-09-06T12:39:10.685 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 69 kB/s | 2.8 kB 00:00 2021-09-06T12:39:12.769 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:12.770 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:39:12.770 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:17.0.0-7472.g0eb1a794.el8.noarch 2021-09-06T12:39:12.770 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.7.0-1.el8.noarch 2021-09-06T12:39:12.770 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.5.1-5.el8.noarch 2021-09-06T12:39:12.771 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-09-06T12:39:12.771 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:12.771 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:39:12.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2021-09-06T12:39:13.258 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:39:13.795 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:39:15.819 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:39:18.540 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:39:18.540 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:18.541 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repository Size 2021-09-06T12:39:18.541 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:18.541 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:39:18.541 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-fuse 2021-09-06T12:39:18.542 INFO:teuthology.orchestra.run.smithi041.stdout: x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 868 k 2021-09-06T12:39:18.542 INFO:teuthology.orchestra.run.smithi041.stdout:Installing dependencies: 2021-09-06T12:39:18.542 INFO:teuthology.orchestra.run.smithi041.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-06T12:39:18.542 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:18.542 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:39:18.543 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:18.543 INFO:teuthology.orchestra.run.smithi041.stdout:Install 2 Packages 2021-09-06T12:39:18.543 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:18.544 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 950 k 2021-09-06T12:39:18.544 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 3.1 M 2021-09-06T12:39:18.545 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:39:18.586 INFO:teuthology.orchestra.run.smithi041.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2021-09-06T12:39:18.782 INFO:teuthology.orchestra.run.smithi041.stdout:(2/2): ceph-fuse-17.0.0-7472.g0eb1a794.el8.x86_ 3.5 MB/s | 868 kB 00:00 2021-09-06T12:39:18.783 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:39:18.783 INFO:teuthology.orchestra.run.smithi041.stdout:Total 3.8 MB/s | 950 kB 00:00 2021-09-06T12:39:18.784 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:39:18.807 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:39:18.807 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:39:18.882 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:39:18.888 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:39:18.893 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:39:19.058 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:39:19.730 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-06T12:39:19.909 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/2 2021-09-06T12:39:20.269 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/2 2021-09-06T12:39:20.269 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/2 2021-09-06T12:39:20.692 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-06T12:39:20.693 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:39:24.214 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:39:26.912 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:39:26.913 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:26.913 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2021-09-06T12:39:26.913 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:26.914 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:39:26.914 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse 2021-09-06T12:39:26.914 INFO:teuthology.orchestra.run.smithi052.stdout: x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 868 k 2021-09-06T12:39:26.914 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2021-09-06T12:39:26.915 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2021-09-06T12:39:26.915 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:26.915 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:39:26.915 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:26.915 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2021-09-06T12:39:26.916 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:26.916 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 950 k 2021-09-06T12:39:26.916 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 3.1 M 2021-09-06T12:39:26.916 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:39:26.957 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2021-09-06T12:39:27.154 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-17.0.0-7472.g0eb1a794.el8.x86_ 3.5 MB/s | 868 kB 00:00 2021-09-06T12:39:27.155 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:39:27.155 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.8 MB/s | 950 kB 00:00 2021-09-06T12:39:27.157 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:39:27.180 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:39:27.180 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:39:27.260 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:39:27.265 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:39:27.435 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:39:27.749 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-09-06T12:39:28.382 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/2 2021-09-06T12:39:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2/2 2021-09-06T12:39:28.744 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/2 2021-09-06T12:39:29.051 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-09-06T12:39:29.051 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:39:31.426 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:38:56 PM UTC. 2021-09-06T12:39:34.055 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:34.056 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:39:34.056 INFO:teuthology.orchestra.run.smithi041.stdout: ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-06T12:39:34.056 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:34.056 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:39:34.190 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install librados-devel 2021-09-06T12:39:34.552 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:39:35.132 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:39:39.882 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:39:04 PM UTC. 2021-09-06T12:39:40.345 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-06T12:39:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:42.437 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:39:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-09-06T12:39:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:42.438 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:39:42.588 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2021-09-06T12:39:42.948 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:39:43.498 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-06T12:39:45.580 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:39:48.274 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:39:48.275 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:48.275 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repo Size 2021-09-06T12:39:48.276 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:48.276 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:39:48.276 INFO:teuthology.orchestra.run.smithi041.stdout: librados-devel x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 125 k 2021-09-06T12:39:48.276 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:48.277 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:39:48.277 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:39:48.277 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:39:48.277 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:39:48.277 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 125 k 2021-09-06T12:39:48.278 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 432 k 2021-09-06T12:39:48.278 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:39:48.528 INFO:teuthology.orchestra.run.smithi041.stdout:librados-devel-17.0.0-7472.g0eb1a794.el8.x86_64 495 kB/s | 125 kB 00:00 2021-09-06T12:39:48.529 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:39:48.529 INFO:teuthology.orchestra.run.smithi041.stdout:Total 492 kB/s | 125 kB 00:00 2021-09-06T12:39:48.530 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:39:48.536 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:39:48.537 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:39:48.597 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:39:48.602 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:39:48.622 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:39:48.731 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:39:48.911 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:49.147 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:49.423 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:49.423 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:39:54.071 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:39:56.780 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:39:56.781 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:56.781 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2021-09-06T12:39:56.782 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:56.782 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:39:56.782 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 125 k 2021-09-06T12:39:56.782 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:56.783 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:39:56.783 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:39:56.783 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:39:56.783 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:39:56.783 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 125 k 2021-09-06T12:39:56.784 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 432 k 2021-09-06T12:39:56.784 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:39:56.990 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-17.0.0-7472.g0eb1a794.el8.x86_64 599 kB/s | 125 kB 00:00 2021-09-06T12:39:56.991 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:39:56.991 INFO:teuthology.orchestra.run.smithi052.stdout:Total 595 kB/s | 125 kB 00:00 2021-09-06T12:39:56.992 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:39:56.998 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:39:56.998 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:39:57.058 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:39:57.063 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:39:57.182 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:39:57.509 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:57.740 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:58.002 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:39:58.003 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:40:00.217 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:39:26 PM UTC. 2021-09-06T12:40:02.856 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:02.856 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:40:02.856 INFO:teuthology.orchestra.run.smithi041.stdout: librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:40:02.856 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:02.857 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:03.043 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install libcephfs2 2021-09-06T12:40:03.403 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:03.983 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:08.681 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:39:34 PM UTC. 2021-09-06T12:40:09.259 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:40:11.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:11.298 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:40:11.298 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:40:11.299 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:11.299 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:40:11.432 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2021-09-06T12:40:11.796 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:12.351 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-06T12:40:14.364 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:40:16.567 INFO:teuthology.orchestra.run.smithi041.stdout:Package libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:17.007 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:17.008 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:40:17.009 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:17.105 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install libcephfs-devel 2021-09-06T12:40:17.464 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:17.566 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-06T12:40:18.043 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:40:18.178 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-06T12:40:20.116 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:20.117 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:40:20.118 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repo Size 2021-09-06T12:40:20.118 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:40:20.118 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:40:20.118 INFO:teuthology.orchestra.run.smithi041.stdout: libcephfs-devel x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 24 k 2021-09-06T12:40:20.118 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:20.119 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:40:20.119 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:40:20.119 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:40:20.119 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:20.119 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 24 k 2021-09-06T12:40:20.120 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 102 k 2021-09-06T12:40:20.120 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:40:20.266 INFO:teuthology.orchestra.run.smithi041.stdout:libcephfs-devel-17.0.0-7472.g0eb1a794.el8.x86_6 162 kB/s | 24 kB 00:00 2021-09-06T12:40:20.267 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:40:20.267 INFO:teuthology.orchestra.run.smithi041.stdout:Total 160 kB/s | 24 kB 00:00 2021-09-06T12:40:20.268 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:40:20.271 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:40:20.271 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:40:20.273 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:40:20.278 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:40:20.342 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:40:20.474 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:20.702 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:20.939 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:20.939 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:40:22.914 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:40:25.225 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:25.661 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:40:25.663 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:40:25.663 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:40:25.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2021-09-06T12:40:26.124 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:26.655 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-06T12:40:26.839 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:40:28.772 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:40:28.773 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:40:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2021-09-06T12:40:28.774 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:40:28.774 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:40:28.774 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 24 k 2021-09-06T12:40:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:28.775 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:40:28.775 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:40:28.775 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:40:28.775 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:28.776 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 24 k 2021-09-06T12:40:28.776 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 102 k 2021-09-06T12:40:28.776 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:40:28.975 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-17.0.0-7472.g0eb1a794.el8.x86_6 119 kB/s | 24 kB 00:00 2021-09-06T12:40:28.976 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:40:28.976 INFO:teuthology.orchestra.run.smithi052.stdout:Total 118 kB/s | 24 kB 00:00 2021-09-06T12:40:28.976 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:40:28.980 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:40:28.980 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:40:28.982 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:40:28.986 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:40:29.037 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:40:29.222 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:29.435 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:29.684 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:40:29.684 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:40:31.796 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:39:55 PM UTC. 2021-09-06T12:40:34.438 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:34.438 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:40:34.438 INFO:teuthology.orchestra.run.smithi041.stdout: libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:40:34.439 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:40:34.439 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:34.562 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install librados2 2021-09-06T12:40:34.922 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:35.503 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:40.482 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:40:03 PM UTC. 2021-09-06T12:40:40.868 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-06T12:40:43.112 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:43.112 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:40:43.112 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:40:43.112 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:40:43.113 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:40:43.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2021-09-06T12:40:43.608 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:44.138 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:40:46.445 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:40:48.705 INFO:teuthology.orchestra.run.smithi041.stdout:Package librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:49.157 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:49.158 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:40:49.159 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:49.272 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install librbd1 2021-09-06T12:40:49.328 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:40:49.630 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:50.202 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:40:50.342 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-06T12:40:51.762 INFO:teuthology.orchestra.run.smithi041.stdout:Package librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:52.207 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:52.208 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:40:52.208 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:52.410 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install python3-rados 2021-09-06T12:40:52.767 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:53.360 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:53.498 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:40:54.914 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:40:54.979 INFO:teuthology.orchestra.run.smithi041.stdout:Package python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:55.415 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:55.416 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:40:55.416 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:55.517 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install python3-rgw 2021-09-06T12:40:55.876 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:56.445 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:56.585 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:40:57.171 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:57.611 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:40:57.612 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:40:57.612 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:40:57.714 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2021-09-06T12:40:58.020 INFO:teuthology.orchestra.run.smithi041.stdout:Package python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:40:58.072 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:58.455 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:40:58.456 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:40:58.457 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:40:58.555 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install python3-cephfs 2021-09-06T12:40:58.620 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:40:58.810 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:58.910 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:40:59.492 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:40:59.630 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 77 kB/s | 2.8 kB 00:00 2021-09-06T12:41:00.287 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:00.732 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:00.733 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:41:00.734 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:00.821 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2021-09-06T12:41:01.052 INFO:teuthology.orchestra.run.smithi041.stdout:Package python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:01.176 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:01.491 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:41:01.493 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:41:01.493 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:41:01.601 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install python3-rbd 2021-09-06T12:41:01.718 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 72 kB/s | 2.8 kB 00:00 2021-09-06T12:41:01.907 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:41:01.959 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:02.543 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-06T12:41:02.683 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 68 kB/s | 2.8 kB 00:00 2021-09-06T12:41:03.386 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:03.828 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:03.829 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:41:03.830 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:03.933 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2021-09-06T12:41:04.169 INFO:teuthology.orchestra.run.smithi041.stdout:Package python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:04.290 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:04.625 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:41:04.627 INFO:teuthology.orchestra.run.smithi041.stdout:Nothing to do. 2021-09-06T12:41:04.627 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:41:04.723 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install rbd-fuse 2021-09-06T12:41:04.828 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:41:05.018 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:41:05.080 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:05.659 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:41:05.795 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 2021-09-06T12:41:06.437 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:06.875 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:06.876 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:41:06.876 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:06.975 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2021-09-06T12:41:07.332 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:07.851 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 78 kB/s | 2.8 kB 00:00 2021-09-06T12:41:07.890 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:41:07.891 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:07.891 INFO:teuthology.orchestra.run.smithi041.stdout: Package Architecture Version Repository Size 2021-09-06T12:41:07.891 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:07.891 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:41:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-fuse x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 93 k 2021-09-06T12:41:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:07.892 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:41:07.892 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:07.893 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:41:07.893 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:07.893 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 93 k 2021-09-06T12:41:07.893 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 281 k 2021-09-06T12:41:07.893 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:41:08.035 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 68 kB/s | 2.4 kB 00:00 2021-09-06T12:41:08.060 INFO:teuthology.orchestra.run.smithi041.stdout:rbd-fuse-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 551 kB/s | 93 kB 00:00 2021-09-06T12:41:08.061 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:41:08.062 INFO:teuthology.orchestra.run.smithi041.stdout:Total 547 kB/s | 93 kB 00:00 2021-09-06T12:41:08.062 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:41:08.074 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:41:08.075 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:41:08.134 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:41:08.138 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:41:08.257 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:41:08.409 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:08.650 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:08.929 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:08.929 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:41:09.561 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:10.015 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:10.016 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:41:10.017 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:10.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2021-09-06T12:41:10.479 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:11.048 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-06T12:41:11.236 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 kB/s | 2.4 kB 00:00 2021-09-06T12:41:12.819 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 is already installed. 2021-09-06T12:41:13.259 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:13.260 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2021-09-06T12:41:13.260 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:13.358 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2021-09-06T12:41:13.716 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:14.251 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:41:14.439 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 64 kB/s | 2.4 kB 00:00 2021-09-06T12:41:16.381 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:16.382 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:16.382 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2021-09-06T12:41:16.382 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:16.382 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:41:16.383 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 93 k 2021-09-06T12:41:16.383 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:16.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:41:16.383 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:16.383 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:41:16.385 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:16.386 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 93 k 2021-09-06T12:41:16.386 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 281 k 2021-09-06T12:41:16.386 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:41:16.552 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 547 kB/s | 93 kB 00:00 2021-09-06T12:41:16.553 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:41:16.553 INFO:teuthology.orchestra.run.smithi052.stdout:Total 543 kB/s | 93 kB 00:00 2021-09-06T12:41:16.554 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:41:16.566 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:41:16.567 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:41:16.628 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:41:16.632 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:41:16.753 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:41:16.932 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:17.170 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:17.408 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:17.408 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:41:19.715 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:40:26 PM UTC. 2021-09-06T12:41:22.332 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:22.332 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:41:22.332 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:41:22.333 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:22.333 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:41:22.475 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install rbd-mirror 2021-09-06T12:41:22.833 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:23.428 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 2021-09-06T12:41:28.271 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:40:35 PM UTC. 2021-09-06T12:41:28.844 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 75 kB/s | 2.8 kB 00:00 2021-09-06T12:41:30.813 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:30.814 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:41:30.814 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:41:30.814 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:30.815 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:41:30.929 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2021-09-06T12:41:31.296 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:31.838 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-06T12:41:34.040 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:41:36.736 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:41:36.737 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:36.737 INFO:teuthology.orchestra.run.smithi041.stdout: Package Arch Version Repo Size 2021-09-06T12:41:36.737 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:36.737 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:41:36.738 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-mirror x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.8 M 2021-09-06T12:41:36.738 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:36.738 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:41:36.738 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:41:36.739 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:41:36.739 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:36.739 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 3.8 M 2021-09-06T12:41:36.739 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 19 M 2021-09-06T12:41:36.740 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:41:37.002 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-06T12:41:37.141 INFO:teuthology.orchestra.run.smithi041.stdout:rbd-mirror-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 9.3 MB/s | 3.8 MB 00:00 2021-09-06T12:41:37.141 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:41:37.142 INFO:teuthology.orchestra.run.smithi041.stdout:Total 9.3 MB/s | 3.8 MB 00:00 2021-09-06T12:41:37.142 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:41:37.157 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:41:37.157 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:41:37.230 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:41:37.234 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:41:37.746 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:41:37.924 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:38.316 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:38.552 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:38.552 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:41:42.631 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:41:45.318 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:41:45.320 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2021-09-06T12:41:45.320 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:45.320 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:41:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 3.8 M 2021-09-06T12:41:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:45.321 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:41:45.321 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:41:45.322 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:41:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:41:45.322 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.8 M 2021-09-06T12:41:45.322 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 19 M 2021-09-06T12:41:45.322 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:41:45.725 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 9.3 MB/s | 3.8 MB 00:00 2021-09-06T12:41:45.726 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:41:45.727 INFO:teuthology.orchestra.run.smithi052.stdout:Total 9.3 MB/s | 3.8 MB 00:00 2021-09-06T12:41:45.727 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:41:45.742 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:41:45.742 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:41:45.817 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:41:45.821 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:41:46.334 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:41:46.523 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:46.910 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:47.156 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:41:47.157 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:41:49.296 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:41:14 PM UTC. 2021-09-06T12:41:51.933 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:51.933 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:41:51.934 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:41:51.934 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:41:51.934 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:41:52.059 DEBUG:teuthology.orchestra.run.smithi041:> sudo yum -y install rbd-nbd 2021-09-06T12:41:52.430 INFO:teuthology.orchestra.run.smithi041.stdout:Updating Subscription Management repositories. 2021-09-06T12:41:53.006 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 kB/s | 2.4 kB 00:00 2021-09-06T12:41:57.868 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:41:22 PM UTC. 2021-09-06T12:41:58.325 INFO:teuthology.orchestra.run.smithi041.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 76 kB/s | 2.8 kB 00:00 2021-09-06T12:42:00.502 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:00.502 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:42:00.503 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:42:00.503 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:00.503 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:42:00.633 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2021-09-06T12:42:00.993 INFO:teuthology.orchestra.run.smithi052.stdout:Updating Subscription Management repositories. 2021-09-06T12:42:01.542 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 74 kB/s | 2.8 kB 00:00 2021-09-06T12:42:03.472 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:42:06.133 INFO:teuthology.orchestra.run.smithi041.stdout:Dependencies resolved. 2021-09-06T12:42:06.134 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:42:06.134 INFO:teuthology.orchestra.run.smithi041.stdout: Package Architecture Version Repository Size 2021-09-06T12:42:06.134 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:42:06.134 INFO:teuthology.orchestra.run.smithi041.stdout:Installing: 2021-09-06T12:42:06.135 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-nbd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 173 k 2021-09-06T12:42:06.135 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:06.135 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction Summary 2021-09-06T12:42:06.135 INFO:teuthology.orchestra.run.smithi041.stdout:================================================================================ 2021-09-06T12:42:06.135 INFO:teuthology.orchestra.run.smithi041.stdout:Install 1 Package 2021-09-06T12:42:06.136 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:06.136 INFO:teuthology.orchestra.run.smithi041.stdout:Total download size: 173 k 2021-09-06T12:42:06.136 INFO:teuthology.orchestra.run.smithi041.stdout:Installed size: 563 k 2021-09-06T12:42:06.136 INFO:teuthology.orchestra.run.smithi041.stdout:Downloading Packages: 2021-09-06T12:42:06.285 INFO:teuthology.orchestra.run.smithi041.stdout:rbd-nbd-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 1.1 MB/s | 173 kB 00:00 2021-09-06T12:42:06.286 INFO:teuthology.orchestra.run.smithi041.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:42:06.287 INFO:teuthology.orchestra.run.smithi041.stdout:Total 1.1 MB/s | 173 kB 00:00 2021-09-06T12:42:06.287 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction check 2021-09-06T12:42:06.300 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction check succeeded. 2021-09-06T12:42:06.300 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction test 2021-09-06T12:42:06.361 INFO:teuthology.orchestra.run.smithi041.stdout:Transaction test succeeded. 2021-09-06T12:42:06.364 INFO:teuthology.orchestra.run.smithi041.stdout:Running transaction 2021-09-06T12:42:06.491 INFO:teuthology.orchestra.run.smithi041.stdout: Preparing : 1/1 2021-09-06T12:42:06.614 INFO:teuthology.orchestra.run.smithi052.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 kB/s | 2.4 kB 00:00 2021-09-06T12:42:06.676 INFO:teuthology.orchestra.run.smithi041.stdout: Installing : rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:06.907 INFO:teuthology.orchestra.run.smithi041.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:07.154 INFO:teuthology.orchestra.run.smithi041.stdout: Verifying : rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:07.155 INFO:teuthology.orchestra.run.smithi041.stdout:Installed products updated. 2021-09-06T12:42:11.962 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:33:20 PM UTC. 2021-09-06T12:42:14.624 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2021-09-06T12:42:14.626 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:42:14.626 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2021-09-06T12:42:14.627 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:42:14.627 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2021-09-06T12:42:14.628 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:17.0.0-7472.g0eb1a794.el8 ceph 173 k 2021-09-06T12:42:14.628 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:14.629 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2021-09-06T12:42:14.629 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2021-09-06T12:42:14.630 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2021-09-06T12:42:14.630 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:14.631 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 173 k 2021-09-06T12:42:14.631 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 563 k 2021-09-06T12:42:14.632 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2021-09-06T12:42:14.797 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-17.0.0-7472.g0eb1a794.el8.x86_64.rpm 1.0 MB/s | 173 kB 00:00 2021-09-06T12:42:14.798 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2021-09-06T12:42:14.798 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.0 MB/s | 173 kB 00:00 2021-09-06T12:42:14.799 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2021-09-06T12:42:14.812 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2021-09-06T12:42:14.812 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2021-09-06T12:42:14.872 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2021-09-06T12:42:14.876 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2021-09-06T12:42:15.003 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2021-09-06T12:42:15.174 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:15.448 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:15.740 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 1/1 2021-09-06T12:42:15.740 INFO:teuthology.orchestra.run.smithi052.stdout:Installed products updated. 2021-09-06T12:42:17.804 INFO:teuthology.orchestra.run.smithi041.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:41:44 PM UTC. 2021-09-06T12:42:20.372 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:20.373 INFO:teuthology.orchestra.run.smithi041.stdout:Installed: 2021-09-06T12:42:20.373 INFO:teuthology.orchestra.run.smithi041.stdout: rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:42:20.373 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:20.373 INFO:teuthology.orchestra.run.smithi041.stdout:Complete! 2021-09-06T12:42:20.488 DEBUG:teuthology.parallel:result is None 2021-09-06T12:42:26.565 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:05 ago on Mon 06 Sep 2021 12:41:52 PM UTC. 2021-09-06T12:42:29.113 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:29.114 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2021-09-06T12:42:29.114 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:17.0.0-7472.g0eb1a794.el8.x86_64 2021-09-06T12:42:29.114 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:29.115 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2021-09-06T12:42:29.273 DEBUG:teuthology.parallel:result is None 2021-09-06T12:42:29.274 DEBUG:teuthology.orchestra.run.smithi041:> sudo lsb_release -is 2021-09-06T12:42:29.322 INFO:teuthology.orchestra.run.smithi041.stdout:RedHatEnterprise 2021-09-06T12:42:29.324 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:42:29.325 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0eb1a7943dd70e2a0b3086ea680284137a187e73 2021-09-06T12:42:29.474 DEBUG:teuthology.orchestra.run.smithi041:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-06T12:42:29.497 INFO:teuthology.orchestra.run.smithi041.stdout:17.0.0-7472.g0eb1a794.el8 2021-09-06T12:42:29.498 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7472.g0eb1a794.el8 2021-09-06T12:42:29.498 INFO:teuthology.task.install:The correct ceph version 17.0.0-7472.g0eb1a794 is installed. 2021-09-06T12:42:29.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-09-06T12:42:29.540 INFO:teuthology.orchestra.run.smithi052.stdout:RedHatEnterprise 2021-09-06T12:42:29.541 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2021-09-06T12:42:29.542 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0eb1a7943dd70e2a0b3086ea680284137a187e73 2021-09-06T12:42:29.686 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-09-06T12:42:29.706 INFO:teuthology.orchestra.run.smithi052.stdout:17.0.0-7472.g0eb1a794.el8 2021-09-06T12:42:29.706 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7472.g0eb1a794.el8 2021-09-06T12:42:29.707 INFO:teuthology.task.install:The correct ceph version 17.0.0-7472.g0eb1a794 is installed. 2021-09-06T12:42:29.707 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-09-06T12:42:29.708 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:29.708 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-06T12:42:29.738 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:29.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-09-06T12:42:29.770 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-09-06T12:42:29.771 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:29.771 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/usr/bin/daemon-helper 2021-09-06T12:42:29.808 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-06T12:42:29.876 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:29.877 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2021-09-06T12:42:29.901 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-09-06T12:42:29.966 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-09-06T12:42:29.966 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:29.967 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-06T12:42:29.994 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-06T12:42:30.061 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:30.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2021-09-06T12:42:30.086 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-09-06T12:42:30.151 INFO:teuthology.run_tasks:Running task ceph... 2021-09-06T12:42:30.212 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-09-06T12:42:30.213 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 777 /var/log/ceph 2021-09-06T12:42:30.216 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2021-09-06T12:42:30.241 INFO:tasks.ceph:Disabling ceph logrotate... 2021-09-06T12:42:30.242 DEBUG:teuthology.orchestra.run.smithi041:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-06T12:42:30.284 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2021-09-06T12:42:30.310 INFO:tasks.ceph:Creating extra log directories... 2021-09-06T12:42:30.310 DEBUG:teuthology.orchestra.run.smithi041:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-06T12:42:30.352 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-09-06T12:42:30.388 INFO:tasks.ceph:Creating ceph cluster c1... 2021-09-06T12:42:30.388 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-06T12:42:30.389 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857', 'branch': 'wip-skoduri-lua', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6376857', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-lua'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-lua', 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-lua', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'targets': {'smithi041.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKEAsKjpnirpPJZlrVlDB6EUcJevmYskuRXgQf6uMRHU+EU2BE/X82jOB1CqMpIfr8BU2Ji7R/5NNG5JscS8FgZ06sUAoAo8g06h6Qf4JVI/JcRlmNOALGPSbdOQVranDh1SQn+Jhs02F/M+I1xdxwd63mXbfaGGEe4YpIuYozFVISgrXDNjEm3BOxUP9BUhWNyzVIFjB4uLvNKqP03Zpp1amOEJ38pXu9IlCC1RMFhSqw7eWQDih957x+Pl8LbUBtQlq4b+XRrkSCL+OwdboW3keev60KLitsQqYpvGlAdynHhRrC0tWiMOmTQa4fEyzf6fZFZVAC9AYG1X7TIiQg/cVokcFCG2iymZBE0LXbzGRulXWiDE4orhmlt818fI+SCJoJ7exNpUpe/7H7bdLDncrY7KVO4fbaK8IGp32jmHMzYdMJ6yRw9Io25LtFFY/Obz0EZW4UpV+ccEpwtuFKR0qjvAQjUdurnqHU7dB5Ujdi4o4Xv/sgr2AO5KEChc=', 'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT1FWubehwPPALY5EhmmXs2T9Kf7zlSb7OkqXs4OzGfyd3xKEh38q10xEE4m6rgXG+GBpWHBJQvZUJSyBukInqY5N7EJ0/nzsyPQt/c7DcWSgvFuE4sA/ihpjg5ZikISU523gDbtc9AXqYX6wUouhh3LJ1ZI25LwHZQCkFON34Sii0I8wn58/9RtZY8AZ8C86UhbYZbVQvYwQJOYVPprf0rXSiQy4A1sZR4J/jm4oVU6VBtkzvzOBZNME011hmU5yWJ/RB6CPaf5nCO82HEBJorqfQ/QnJZeq3m5Jf87+mnOtitKe/ml8M6a0fcBq6XnGvPoD4o9bzRtrNg478pVxFOU+dbOI/YaV7zAtIWER2V5LRZgDzdmoFXBBxABf3h9PSiX3yPvT3PGbeNHAQctnGqLo1fYhBBKMt2mfxFkq/J5K+7GY8Z/Z1IlxE1qB479mOS+AiNxV6VwljlP5WIA2EkKjRAEvNjwr+xh12V2jzSNzH/IAYZ8R4/VfkmE6MUR8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-06_10:30:05', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-06T12:42:30.389 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-06T12:42:30.419 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-09-06T12:42:30.434 DEBUG:teuthology.orchestra.run.smithi041:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T12:42:30.475 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T12:42:30.500 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:30.500 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/scratch_devs of=/dev/stdout 2021-09-06T12:42:30.554 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-06T12:42:30.555 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-06T12:42:30.610 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-06T12:42:30.610 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:30.610 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 110449 Links: 1 2021-09-06T12:42:30.611 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:30.611 INFO:teuthology.orchestra.run.smithi041.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:30.611 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-09-06 12:41:38.126055071 +0000 2021-09-06T12:42:30.611 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-09-06 12:32:18.760674134 +0000 2021-09-06T12:42:30.612 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-09-06 12:32:18.760674134 +0000 2021-09-06T12:42:30.612 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-09-06T12:42:30.620 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-09-06T12:42:30.621 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-09-06T12:42:30.621 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000241058 s, 2.1 MB/s 2021-09-06T12:42:30.624 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-06T12:42:30.678 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-06T12:42:30.679 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:30.679 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 115121 Links: 1 2021-09-06T12:42:30.679 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:30.680 INFO:teuthology.orchestra.run.smithi041.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:30.680 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-09-06 12:41:38.126055071 +0000 2021-09-06T12:42:30.680 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-09-06 12:32:19.097668886 +0000 2021-09-06T12:42:30.680 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-09-06 12:32:19.097668886 +0000 2021-09-06T12:42:30.680 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-09-06T12:42:30.689 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-09-06T12:42:30.689 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-09-06T12:42:30.689 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000211092 s, 2.4 MB/s 2021-09-06T12:42:30.692 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-06T12:42:30.746 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-06T12:42:30.747 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:30.747 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 115151 Links: 1 2021-09-06T12:42:30.747 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:30.747 INFO:teuthology.orchestra.run.smithi041.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:30.748 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-09-06 12:41:38.126055071 +0000 2021-09-06T12:42:30.748 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-09-06 12:32:19.454663326 +0000 2021-09-06T12:42:30.748 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-09-06 12:32:19.454663326 +0000 2021-09-06T12:42:30.748 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-09-06T12:42:30.756 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-09-06T12:42:30.757 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-09-06T12:42:30.757 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.000211099 s, 2.4 MB/s 2021-09-06T12:42:30.759 DEBUG:teuthology.orchestra.run.smithi041:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-06T12:42:30.814 INFO:teuthology.orchestra.run.smithi041.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-06T12:42:30.815 INFO:teuthology.orchestra.run.smithi041.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:30.815 INFO:teuthology.orchestra.run.smithi041.stdout:Device: 6h/6d Inode: 112146 Links: 1 2021-09-06T12:42:30.815 INFO:teuthology.orchestra.run.smithi041.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:30.815 INFO:teuthology.orchestra.run.smithi041.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:30.816 INFO:teuthology.orchestra.run.smithi041.stdout:Access: 2021-09-06 12:41:38.126055071 +0000 2021-09-06T12:42:30.816 INFO:teuthology.orchestra.run.smithi041.stdout:Modify: 2021-09-06 12:32:19.821657612 +0000 2021-09-06T12:42:30.816 INFO:teuthology.orchestra.run.smithi041.stdout:Change: 2021-09-06 12:32:19.821657612 +0000 2021-09-06T12:42:30.816 INFO:teuthology.orchestra.run.smithi041.stdout: Birth: - 2021-09-06T12:42:30.824 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records in 2021-09-06T12:42:30.824 INFO:teuthology.orchestra.run.smithi041.stderr:1+0 records out 2021-09-06T12:42:30.825 INFO:teuthology.orchestra.run.smithi041.stderr:512 bytes copied, 0.00019618 s, 2.6 MB/s 2021-09-06T12:42:30.827 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:30.827 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi041.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T12:42:30.828 INFO:tasks.ceph:Generating config... 2021-09-06T12:42:30.829 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-06T12:42:30.829 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-06T12:42:30.830 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-06T12:42:30.830 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:30.830 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-06T12:42:30.831 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-06T12:42:30.831 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-06T12:42:30.831 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-06T12:42:30.831 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-06T12:42:30.832 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-06T12:42:30.832 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-06T12:42:30.832 INFO:tasks.ceph:[client] setuser = ceph 2021-09-06T12:42:30.833 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-06T12:42:30.833 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-06T12:42:30.833 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-06T12:42:30.834 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-06T12:42:30.834 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-06T12:42:30.834 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-06T12:42:30.835 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-06T12:42:30.835 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-06T12:42:30.835 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-06T12:42:30.836 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-06T12:42:30.836 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-06T12:42:30.836 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:30.837 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:30.837 INFO:tasks.ceph:Setting up c1.mon.a... 2021-09-06T12:42:30.837 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-09-06T12:42:30.921 INFO:teuthology.orchestra.run.smithi041.stdout:creating /etc/ceph/c1.keyring 2021-09-06T12:42:30.923 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-09-06T12:42:30.963 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T12:42:31.026 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.41')] 2021-09-06T12:42:31.027 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.41'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-06T12:42:31.028 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:31.029 DEBUG:teuthology.orchestra.run.smithi041:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T12:42:31.081 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.41 --print /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:31.156 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:42:31.157 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:42:31.158 INFO:teuthology.orchestra.run.smithi041.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:31.159 INFO:teuthology.orchestra.run.smithi041.stdout:monmaptool: generated fsid c2315c9e-41d7-497d-bdb5-f190256454f1 2021-09-06T12:42:31.159 INFO:teuthology.orchestra.run.smithi041.stdout:setting min_mon_release = octopus 2021-09-06T12:42:31.159 INFO:teuthology.orchestra.run.smithi041.stdout:epoch 0 2021-09-06T12:42:31.159 INFO:teuthology.orchestra.run.smithi041.stdout:fsid c2315c9e-41d7-497d-bdb5-f190256454f1 2021-09-06T12:42:31.160 INFO:teuthology.orchestra.run.smithi041.stdout:last_changed 2021-09-06T12:42:31.159068+0000 2021-09-06T12:42:31.160 INFO:teuthology.orchestra.run.smithi041.stdout:created 2021-09-06T12:42:31.159068+0000 2021-09-06T12:42:31.160 INFO:teuthology.orchestra.run.smithi041.stdout:min_mon_release 15 (octopus) 2021-09-06T12:42:31.160 INFO:teuthology.orchestra.run.smithi041.stdout:election_strategy: 1 2021-09-06T12:42:31.161 INFO:teuthology.orchestra.run.smithi041.stdout:0: [v2:172.21.15.41:3300/0,v1:172.21.15.41:6789/0] mon.a 2021-09-06T12:42:31.161 INFO:teuthology.orchestra.run.smithi041.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-09-06T12:42:31.162 DEBUG:teuthology.orchestra.run.smithi041:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T12:42:31.213 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID c2315c9e-41d7-497d-bdb5-f190256454f1... 2021-09-06T12:42:31.214 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-06T12:42:31.256 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-09-06T12:42:31.296 INFO:teuthology.orchestra.run.smithi041.stdout:[global] 2021-09-06T12:42:31.297 INFO:teuthology.orchestra.run.smithi041.stdout: chdir = "" 2021-09-06T12:42:31.297 INFO:teuthology.orchestra.run.smithi041.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T12:42:31.297 INFO:teuthology.orchestra.run.smithi041.stdout: auth supported = cephx 2021-09-06T12:42:31.298 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.298 INFO:teuthology.orchestra.run.smithi041.stdout: filestore xattr use omap = true 2021-09-06T12:42:31.298 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.298 INFO:teuthology.orchestra.run.smithi041.stdout: mon clock drift allowed = 1.000 2021-09-06T12:42:31.299 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.299 INFO:teuthology.orchestra.run.smithi041.stdout: osd crush chooseleaf type = 0 2021-09-06T12:42:31.299 INFO:teuthology.orchestra.run.smithi041.stdout: auth debug = true 2021-09-06T12:42:31.299 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.300 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on old message = true 2021-09-06T12:42:31.300 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on bug = true 2021-09-06T12:42:31.300 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.300 INFO:teuthology.orchestra.run.smithi041.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T12:42:31.301 INFO:teuthology.orchestra.run.smithi041.stdout: mon pg warn max object skew = 0 2021-09-06T12:42:31.301 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.301 INFO:teuthology.orchestra.run.smithi041.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T12:42:31.301 INFO:teuthology.orchestra.run.smithi041.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T12:42:31.301 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.302 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default size = 2 2021-09-06T12:42:31.302 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.302 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow primary affinity = true 2021-09-06T12:42:31.302 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow pg remap = true 2021-09-06T12:42:31.302 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on legacy crush tunables = false 2021-09-06T12:42:31.303 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on crush straw calc version zero = false 2021-09-06T12:42:31.303 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on no sortbitwise = false 2021-09-06T12:42:31.303 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on osd down out interval zero = false 2021-09-06T12:42:31.303 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on too few osds = false 2021-09-06T12:42:31.304 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T12:42:31.304 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T12:42:31.304 INFO:teuthology.orchestra.run.smithi041.stdout: mon_allow_pool_size_one = true 2021-09-06T12:42:31.304 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.304 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T12:42:31.305 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.305 INFO:teuthology.orchestra.run.smithi041.stdout: osd default data pool replay window = 5 2021-09-06T12:42:31.305 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.305 INFO:teuthology.orchestra.run.smithi041.stdout: mon allow pool delete = true 2021-09-06T12:42:31.306 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.306 INFO:teuthology.orchestra.run.smithi041.stdout: mon cluster log file level = debug 2021-09-06T12:42:31.306 INFO:teuthology.orchestra.run.smithi041.stdout: debug asserts on shutdown = true 2021-09-06T12:42:31.306 INFO:teuthology.orchestra.run.smithi041.stdout: mon health detail to clog = false 2021-09-06T12:42:31.307 INFO:teuthology.orchestra.run.smithi041.stdout: mon host = 172.21.15.41 2021-09-06T12:42:31.307 INFO:teuthology.orchestra.run.smithi041.stdout: fsid = c2315c9e-41d7-497d-bdb5-f190256454f1 2021-09-06T12:42:31.307 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.307 INFO:teuthology.orchestra.run.smithi041.stdout:[osd] 2021-09-06T12:42:31.307 INFO:teuthology.orchestra.run.smithi041.stdout: osd journal size = 100 2021-09-06T12:42:31.308 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.308 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub load threshold = 5.0 2021-09-06T12:42:31.308 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub max interval = 600 2021-09-06T12:42:31.308 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.309 INFO:teuthology.orchestra.run.smithi041.stdout: osd recover clone overlap = true 2021-09-06T12:42:31.309 INFO:teuthology.orchestra.run.smithi041.stdout: osd recovery max chunk = 1048576 2021-09-06T12:42:31.309 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.309 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug shutdown = true 2021-09-06T12:42:31.309 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug op order = true 2021-09-06T12:42:31.310 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug verify stray on activate = true 2021-09-06T12:42:31.310 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.310 INFO:teuthology.orchestra.run.smithi041.stdout: osd open classes on start = true 2021-09-06T12:42:31.310 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug pg log writeout = true 2021-09-06T12:42:31.311 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.311 INFO:teuthology.orchestra.run.smithi041.stdout: osd deep scrub update digest min age = 30 2021-09-06T12:42:31.311 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.311 INFO:teuthology.orchestra.run.smithi041.stdout: osd map max advance = 10 2021-09-06T12:42:31.311 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.312 INFO:teuthology.orchestra.run.smithi041.stdout: journal zero on create = true 2021-09-06T12:42:31.312 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.312 INFO:teuthology.orchestra.run.smithi041.stdout: filestore ondisk finisher threads = 3 2021-09-06T12:42:31.312 INFO:teuthology.orchestra.run.smithi041.stdout: filestore apply finisher threads = 3 2021-09-06T12:42:31.313 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.313 INFO:teuthology.orchestra.run.smithi041.stdout: bdev debug aio = true 2021-09-06T12:42:31.313 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug misdirected ops = true 2021-09-06T12:42:31.313 INFO:teuthology.orchestra.run.smithi041.stdout: debug filestore = 20 2021-09-06T12:42:31.313 INFO:teuthology.orchestra.run.smithi041.stdout: debug journal = 20 2021-09-06T12:42:31.314 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 20 2021-09-06T12:42:31.314 INFO:teuthology.orchestra.run.smithi041.stdout: debug osd = 25 2021-09-06T12:42:31.314 INFO:teuthology.orchestra.run.smithi041.stdout: rocksdb delete range threshold = 0 2021-09-06T12:42:31.314 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.315 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr] 2021-09-06T12:42:31.315 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-09-06T12:42:31.315 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgr = 20 2021-09-06T12:42:31.315 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-09-06T12:42:31.316 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-09-06T12:42:31.316 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:31.316 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:31.316 INFO:teuthology.orchestra.run.smithi041.stdout: mgr/telemetry/nag = false 2021-09-06T12:42:31.316 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.317 INFO:teuthology.orchestra.run.smithi041.stdout:[mon] 2021-09-06T12:42:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-09-06T12:42:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-09-06T12:42:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: debug paxos = 20 2021-09-06T12:42:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-09-06T12:42:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: mon data avail warn = 5 2021-09-06T12:42:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: mon mgr mkfs grace = 240 2021-09-06T12:42:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd reporter subtree level = osd 2021-09-06T12:42:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd prime pg temp = true 2021-09-06T12:42:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T12:42:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T12:42:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T12:42:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T12:42:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T12:42:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T12:42:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:31.321 INFO:teuthology.orchestra.run.smithi041.stdout:[client] 2021-09-06T12:42:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: rgw cache enabled = true 2021-09-06T12:42:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable ops log = true 2021-09-06T12:42:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable usage log = true 2021-09-06T12:42:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T12:42:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T12:42:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: debug rgw = 20 2021-09-06T12:42:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt require ssl = False 2021-09-06T12:42:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt s3 kms backend = testing 2021-09-06T12:42:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: rgw curl low speed time = 300 2021-09-06T12:42:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: rgw data log num shards = 4 2021-09-06T12:42:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: rgw md log max shards = 4 2021-09-06T12:42:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync log trim interval = 0 2021-09-06T12:42:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T12:42:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync obj etag verify = True 2021-09-06T12:42:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: setgroup = ceph 2021-09-06T12:42:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: setuser = ceph 2021-09-06T12:42:31.325 INFO:teuthology.orchestra.run.smithi041.stdout:[mon.a] 2021-09-06T12:42:31.325 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.0] 2021-09-06T12:42:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T12:42:31.326 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.1] 2021-09-06T12:42:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:31.326 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.2] 2021-09-06T12:42:31.327 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:31.328 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2021-09-06T12:42:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2021-09-06T12:42:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T12:42:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2021-09-06T12:42:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2021-09-06T12:42:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2021-09-06T12:42:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2021-09-06T12:42:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2021-09-06T12:42:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2021-09-06T12:42:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2021-09-06T12:42:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T12:42:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2021-09-06T12:42:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T12:42:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T12:42:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2021-09-06T12:42:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2021-09-06T12:42:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2021-09-06T12:42:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2021-09-06T12:42:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2021-09-06T12:42:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2021-09-06T12:42:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2021-09-06T12:42:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2021-09-06T12:42:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T12:42:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T12:42:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2021-09-06T12:42:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T12:42:31.336 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.337 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2021-09-06T12:42:31.337 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.337 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2021-09-06T12:42:31.337 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.337 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2021-09-06T12:42:31.338 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2021-09-06T12:42:31.338 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2021-09-06T12:42:31.338 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.41 2021-09-06T12:42:31.338 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = c2315c9e-41d7-497d-bdb5-f190256454f1 2021-09-06T12:42:31.339 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.339 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2021-09-06T12:42:31.339 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2021-09-06T12:42:31.339 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.340 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2021-09-06T12:42:31.340 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2021-09-06T12:42:31.340 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.340 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2021-09-06T12:42:31.340 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2021-09-06T12:42:31.341 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.341 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2021-09-06T12:42:31.341 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2021-09-06T12:42:31.341 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2021-09-06T12:42:31.342 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.342 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2021-09-06T12:42:31.342 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2021-09-06T12:42:31.342 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.342 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2021-09-06T12:42:31.343 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.343 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2021-09-06T12:42:31.343 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.343 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2021-09-06T12:42:31.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.344 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2021-09-06T12:42:31.344 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2021-09-06T12:42:31.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.344 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2021-09-06T12:42:31.345 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2021-09-06T12:42:31.345 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 20 2021-09-06T12:42:31.345 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 20 2021-09-06T12:42:31.345 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 20 2021-09-06T12:42:31.346 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 25 2021-09-06T12:42:31.346 INFO:teuthology.orchestra.run.smithi052.stdout: rocksdb delete range threshold = 0 2021-09-06T12:42:31.346 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.346 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2021-09-06T12:42:31.346 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-09-06T12:42:31.347 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2021-09-06T12:42:31.347 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-09-06T12:42:31.347 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-09-06T12:42:31.347 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:31.348 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:31.348 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2021-09-06T12:42:31.348 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.348 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2021-09-06T12:42:31.349 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-09-06T12:42:31.349 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-09-06T12:42:31.349 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2021-09-06T12:42:31.349 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-09-06T12:42:31.349 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2021-09-06T12:42:31.350 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2021-09-06T12:42:31.350 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:31.350 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2021-09-06T12:42:31.350 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2021-09-06T12:42:31.351 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:31.351 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.351 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T12:42:31.351 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T12:42:31.351 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T12:42:31.352 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.352 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T12:42:31.352 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T12:42:31.352 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T12:42:31.353 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:31.353 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2021-09-06T12:42:31.353 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2021-09-06T12:42:31.353 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2021-09-06T12:42:31.353 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2021-09-06T12:42:31.354 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T12:42:31.354 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T12:42:31.354 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2021-09-06T12:42:31.354 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2021-09-06T12:42:31.355 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2021-09-06T12:42:31.355 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:31.355 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2021-09-06T12:42:31.355 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2021-09-06T12:42:31.355 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2021-09-06T12:42:31.356 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2021-09-06T12:42:31.356 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T12:42:31.356 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2021-09-06T12:42:31.356 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2021-09-06T12:42:31.357 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2021-09-06T12:42:31.357 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2021-09-06T12:42:31.357 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2021-09-06T12:42:31.357 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T12:42:31.358 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2021-09-06T12:42:31.358 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:31.358 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2021-09-06T12:42:31.358 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:31.359 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-09-06T12:42:31.359 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-09-06T12:42:31.419 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-06T12:42:31.419 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:31.420 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-09-06T12:42:31.434 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:31.435 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-09-06T12:42:31.489 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:42:31.489 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:31.490 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/ceph/c1.keyring 2021-09-06T12:42:31.490 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T12:42:31.563 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:31.564 DEBUG:teuthology.orchestra.run.smithi041:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:31.618 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:42:31.618 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:31.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/c1.keyring 2021-09-06T12:42:31.619 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-09-06T12:42:31.659 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:31.660 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:31.716 INFO:tasks.ceph:Setting up mon nodes... 2021-09-06T12:42:31.716 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-06T12:42:31.717 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-09-06T12:42:31.765 INFO:teuthology.orchestra.run.smithi041.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-09-06T12:42:31.767 INFO:tasks.ceph:Setting up mds nodes... 2021-09-06T12:42:31.768 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-06T12:42:31.768 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-09-06T12:42:31.806 INFO:teuthology.orchestra.run.smithi041.stdout:creating /etc/ceph/c1.client.0.keyring 2021-09-06T12:42:31.818 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-09-06T12:42:31.896 INFO:teuthology.orchestra.run.smithi041.stdout:creating /etc/ceph/c1.client.1.keyring 2021-09-06T12:42:31.908 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-06T12:42:31.909 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi041.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T12:42:31.909 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-09-06T12:42:31.972 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:31.973 INFO:tasks.ceph:role: c1.osd.0 2021-09-06T12:42:31.973 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:42:31.973 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-06T12:42:32.044 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:32.044 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:32.045 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:32.045 INFO:teuthology.orchestra.run.smithi041.stdout: = reflink=1 2021-09-06T12:42:32.045 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:32.045 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:32.046 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:32.046 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:32.046 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:32.046 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:32.411 INFO:teuthology.orchestra.run.smithi041.stdout:Discarding blocks...Done. 2021-09-06T12:42:32.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-09-06T12:42:32.413 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-09-06T12:42:32.550 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-09-06T12:42:32.580 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-09-06T12:42:32.646 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:32.646 INFO:tasks.ceph:role: c1.osd.1 2021-09-06T12:42:32.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:42:32.647 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-06T12:42:32.717 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:32.719 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:32.720 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:32.720 INFO:teuthology.orchestra.run.smithi041.stdout: = reflink=1 2021-09-06T12:42:32.720 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:32.720 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:32.721 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:32.721 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:32.721 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:32.721 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:33.077 INFO:teuthology.orchestra.run.smithi041.stdout:Discarding blocks...Done. 2021-09-06T12:42:33.078 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-09-06T12:42:33.079 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-09-06T12:42:33.165 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-09-06T12:42:33.194 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-09-06T12:42:33.259 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:33.260 INFO:tasks.ceph:role: c1.osd.2 2021-09-06T12:42:33.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:42:33.260 DEBUG:teuthology.orchestra.run.smithi041:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-06T12:42:33.331 INFO:teuthology.orchestra.run.smithi041.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:33.332 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:33.332 INFO:teuthology.orchestra.run.smithi041.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:33.332 INFO:teuthology.orchestra.run.smithi041.stdout: = reflink=1 2021-09-06T12:42:33.332 INFO:teuthology.orchestra.run.smithi041.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:33.333 INFO:teuthology.orchestra.run.smithi041.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:33.333 INFO:teuthology.orchestra.run.smithi041.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:33.333 INFO:teuthology.orchestra.run.smithi041.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:33.333 INFO:teuthology.orchestra.run.smithi041.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:33.333 INFO:teuthology.orchestra.run.smithi041.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:33.696 INFO:teuthology.orchestra.run.smithi041.stdout:Discarding blocks...Done. 2021-09-06T12:42:33.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi041.front.sepia.ceph.com -o noatime 2021-09-06T12:42:33.698 DEBUG:teuthology.orchestra.run.smithi041:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-09-06T12:42:33.803 DEBUG:teuthology.orchestra.run.smithi041:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-09-06T12:42:33.832 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:33.915 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.916+0000 7fa9821700c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-09-06T12:42:33.916 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.917+0000 7fa9821700c0 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-09-06T12:42:33.916 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.917+0000 7fa9821700c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T12:42:33.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.917+0000 7fa9821700c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T12:42:33.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.917+0000 7fa9821700c0 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-09-06T12:42:33.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:33.919+0000 7fa9821700c0 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-09-06T12:42:35.999 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-09-06T12:42:36.069 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:36.157 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.157+0000 7ff74e1890c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-09-06T12:42:36.157 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.158+0000 7ff74e1890c0 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-09-06T12:42:36.157 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.158+0000 7ff74e1890c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T12:42:36.158 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.158+0000 7ff74e1890c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T12:42:36.158 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.158+0000 7ff74e1890c0 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-09-06T12:42:36.158 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:36.158+0000 7ff74e1890c0 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-09-06T12:42:38.238 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-09-06T12:42:38.268 DEBUG:teuthology.orchestra.run.smithi041:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:38.355 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.356+0000 7f179bda20c0 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-09-06T12:42:38.355 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.356+0000 7f179bda20c0 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-09-06T12:42:38.355 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.356+0000 7f179bda20c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T12:42:38.356 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.356+0000 7f179bda20c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T12:42:38.356 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.356+0000 7f179bda20c0 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-09-06T12:42:38.356 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:42:38.357+0000 7f179bda20c0 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-09-06T12:42:40.436 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-09-06T12:42:40.466 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-06T12:42:40.467 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.467 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-09-06T12:42:40.535 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.535 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-09-06T12:42:40.604 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.604 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-09-06T12:42:40.671 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.672 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-09-06T12:42:40.738 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.739 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-09-06T12:42:40.794 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:40.795 DEBUG:teuthology.orchestra.run.smithi041:> dd if=/etc/ceph/c1.client.1.keyring of=/dev/stdout 2021-09-06T12:42:40.850 INFO:tasks.ceph:Adding keys to all mons... 2021-09-06T12:42:40.851 DEBUG:teuthology.orchestra.run.smithi041:> sudo tee -a /etc/ceph/c1.keyring 2021-09-06T12:42:40.915 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr.x] 2021-09-06T12:42:40.916 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC3DDZhRyq+LRAAmcHkxuC30b2C+J5Rfj6vFg== 2021-09-06T12:42:40.916 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.0] 2021-09-06T12:42:40.916 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC5DDZhonSuNhAAZMeyrubKR3wCBqc8k8YvRA== 2021-09-06T12:42:40.916 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.1] 2021-09-06T12:42:40.917 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC8DDZhcFFxCRAAnOfPdsNYoyz39zgH1d8AaA== 2021-09-06T12:42:40.917 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.2] 2021-09-06T12:42:40.917 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC+DDZhhNNEFRAAj3BkQm0lr/CDhsW0l2gIGQ== 2021-09-06T12:42:40.917 INFO:teuthology.orchestra.run.smithi041.stdout:[client.0] 2021-09-06T12:42:40.917 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC3DDZhQsUoMBAAiwrouUcTxtOk8Wqi7zEO9A== 2021-09-06T12:42:40.918 INFO:teuthology.orchestra.run.smithi041.stdout:[client.1] 2021-09-06T12:42:40.918 INFO:teuthology.orchestra.run.smithi041.stdout: key = AQC3DDZhQAmMNRAAemQ+7m4x7SMQkeMIc5tNHA== 2021-09-06T12:42:40.919 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-06T12:42:41.010 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:42:41.059 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:42:41.106 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:42:41.153 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T12:42:41.204 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T12:42:41.254 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-06T12:42:41.255 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-09-06T12:42:41.280 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-09-06T12:42:41.372 DEBUG:teuthology.orchestra.run.smithi041:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-09-06T12:42:41.397 DEBUG:teuthology.orchestra.run.smithi041:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-09-06T12:42:41.452 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-09-06T12:42:41.452 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-09-06T12:42:41.453 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-09-06T12:42:41.494 INFO:tasks.ceph.c1.mon.a:Started 2021-09-06T12:42:41.494 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-09-06T12:42:41.495 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-09-06T12:42:41.495 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-09-06T12:42:41.497 INFO:tasks.ceph.c1.mgr.x:Started 2021-09-06T12:42:41.498 INFO:tasks.ceph:Setting crush tunables to default 2021-09-06T12:42:41.498 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd crush tunables default 2021-09-06T12:42:41.703 INFO:teuthology.orchestra.run.smithi041.stderr:adjusted tunables profile to default 2021-09-06T12:42:41.713 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-09-06T12:42:41.714 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:41.714 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-09-06T12:42:41.738 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:41.739 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-09-06T12:42:41.803 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:41.804 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-09-06T12:42:41.869 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd new 81881c5c-becb-423b-89a0-1ecf30df70f2 0 2021-09-06T12:42:42.117 INFO:teuthology.orchestra.run.smithi041.stdout:0 2021-09-06T12:42:42.129 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd new ba135a35-84bc-400f-89ec-a1d383f0a26f 1 2021-09-06T12:42:42.338 INFO:teuthology.orchestra.run.smithi041.stdout:1 2021-09-06T12:42:42.351 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd new d6e49e26-6b08-4358-b1bb-c510ac35f34d 2 2021-09-06T12:42:42.560 INFO:teuthology.orchestra.run.smithi041.stdout:2 2021-09-06T12:42:42.572 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-09-06T12:42:42.573 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-09-06T12:42:42.575 INFO:tasks.ceph.c1.osd.0:Started 2021-09-06T12:42:42.575 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-09-06T12:42:42.576 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-09-06T12:42:42.577 INFO:tasks.ceph.c1.osd.1:Started 2021-09-06T12:42:42.578 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-09-06T12:42:42.578 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-09-06T12:42:42.580 INFO:tasks.ceph.c1.osd.2:Started 2021-09-06T12:42:42.581 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-06T12:42:42.884 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-06T12:42:43.067 INFO:teuthology.misc.health.smithi041.stdout: 2021-09-06T12:42:43.068 INFO:teuthology.misc.health.smithi041.stdout:{"epoch":5,"fsid":"c2315c9e-41d7-497d-bdb5-f190256454f1","created":"2021-09-06T12:42:41.607676+0000","modified":"2021-09-06T12:42:42.559299+0000","last_up_change":"0.000000","last_in_change":"2021-09-06T12:42:42.559299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"81881c5c-becb-423b-89a0-1ecf30df70f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ba135a35-84bc-400f-89ec-a1d383f0a26f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d6e49e26-6b08-4358-b1bb-c510ac35f34d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:42:43.079 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-06T12:42:43.174 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:43.175+0000 7fd7c97850c0 -1 Falling back to public interface 2021-09-06T12:42:43.175 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T12:42:43.175+0000 7f1ba88290c0 -1 Falling back to public interface 2021-09-06T12:42:43.176 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:43.176+0000 7fd7c394a700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2021-09-06T12:42:43.176 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:43.176+0000 7fd7c394a700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6806/0: (98) Address already in use 2021-09-06T12:42:43.178 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T12:42:43.179+0000 7fbbcf5790c0 -1 Falling back to public interface 2021-09-06T12:42:45.677 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T12:42:45.678+0000 7fbbcf5790c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-06T12:42:45.933 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:45.934+0000 7fd7c97850c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-06T12:42:45.940 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T12:42:45.941+0000 7f1ba88290c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-06T12:42:47.665 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T12:42:47.665+0000 7fbbc5f37700 -1 osd.2 0 waiting for initial osdmap 2021-09-06T12:42:47.666 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T12:42:47.667+0000 7fbbbe8ca700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:42:48.575 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:48.576+0000 7fd7c0143700 -1 osd.0 0 waiting for initial osdmap 2021-09-06T12:42:48.577 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T12:42:48.578+0000 7fd7b8ad6700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:42:48.922 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T12:42:48.922+0000 7f1b9f1e7700 -1 osd.1 0 waiting for initial osdmap 2021-09-06T12:42:48.923 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T12:42:48.924+0000 7f1b97b7a700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:42:49.382 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-09-06T12:42:49.683 INFO:teuthology.misc.health.smithi041.stdout: 2021-09-06T12:42:49.683 INFO:teuthology.misc.health.smithi041.stdout:{"epoch":9,"fsid":"c2315c9e-41d7-497d-bdb5-f190256454f1","created":"2021-09-06T12:42:41.607676+0000","modified":"2021-09-06T12:42:49.575111+0000","last_up_change":"2021-09-06T12:42:49.575111+0000","last_in_change":"2021-09-06T12:42:42.559299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"81881c5c-becb-423b-89a0-1ecf30df70f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6800","nonce":31662},{"type":"v1","addr":"172.21.15.41:6802","nonce":31662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6804","nonce":31662},{"type":"v1","addr":"172.21.15.41:6807","nonce":31662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6813","nonce":31662},{"type":"v1","addr":"172.21.15.41:6815","nonce":31662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6809","nonce":31662},{"type":"v1","addr":"172.21.15.41:6811","nonce":31662}]},"public_addr":"172.21.15.41:6802/31662","cluster_addr":"172.21.15.41:6807/31662","heartbeat_back_addr":"172.21.15.41:6815/31662","heartbeat_front_addr":"172.21.15.41:6811/31662","state":["exists","up"]},{"osd":1,"uuid":"ba135a35-84bc-400f-89ec-a1d383f0a26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6801","nonce":31664},{"type":"v1","addr":"172.21.15.41:6803","nonce":31664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6805","nonce":31664},{"type":"v1","addr":"172.21.15.41:6806","nonce":31664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6812","nonce":31664},{"type":"v1","addr":"172.21.15.41:6814","nonce":31664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6808","nonce":31664},{"type":"v1","addr":"172.21.15.41:6810","nonce":31664}]},"public_addr":"172.21.15.41:6803/31664","cluster_addr":"172.21.15.41:6806/31664","heartbeat_back_addr":"172.21.15.41:6814/31664","heartbeat_front_addr":"172.21.15.41:6810/31664","state":["exists","up"]},{"osd":2,"uuid":"d6e49e26-6b08-4358-b1bb-c510ac35f34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6816","nonce":31665},{"type":"v1","addr":"172.21.15.41:6817","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6818","nonce":31665},{"type":"v1","addr":"172.21.15.41:6819","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6822","nonce":31665},{"type":"v1","addr":"172.21.15.41:6823","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6820","nonce":31665},{"type":"v1","addr":"172.21.15.41:6821","nonce":31665}]},"public_addr":"172.21.15.41:6817/31665","cluster_addr":"172.21.15.41:6819/31665","heartbeat_back_addr":"172.21.15.41:6823/31665","heartbeat_front_addr":"172.21.15.41:6821/31665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.632428+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:42:49.695 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-06T12:42:49.695 INFO:tasks.ceph:Creating RBD pool 2021-09-06T12:42:49.696 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd pool create rbd 8 2021-09-06T12:42:50.678 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'rbd' created 2021-09-06T12:42:50.691 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-06T12:42:51.693 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-06T12:42:51.706 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-09-06T12:42:51.707 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T12:42:51.708 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-06T12:42:52.018 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:52.019 INFO:teuthology.orchestra.run.smithi041.stdout:{"epoch":11,"fsid":"c2315c9e-41d7-497d-bdb5-f190256454f1","created":"2021-09-06T12:42:41.607676+0000","modified":"2021-09-06T12:42:51.675064+0000","last_up_change":"2021-09-06T12:42:49.575111+0000","last_in_change":"2021-09-06T12:42:42.559299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T12:42:50.023827+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T12:42:50.736568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"81881c5c-becb-423b-89a0-1ecf30df70f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6800","nonce":31662},{"type":"v1","addr":"172.21.15.41:6802","nonce":31662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6804","nonce":31662},{"type":"v1","addr":"172.21.15.41:6807","nonce":31662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6813","nonce":31662},{"type":"v1","addr":"172.21.15.41:6815","nonce":31662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6809","nonce":31662},{"type":"v1","addr":"172.21.15.41:6811","nonce":31662}]},"public_addr":"172.21.15.41:6802/31662","cluster_addr":"172.21.15.41:6807/31662","heartbeat_back_addr":"172.21.15.41:6815/31662","heartbeat_front_addr":"172.21.15.41:6811/31662","state":["exists","up"]},{"osd":1,"uuid":"ba135a35-84bc-400f-89ec-a1d383f0a26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6801","nonce":31664},{"type":"v1","addr":"172.21.15.41:6803","nonce":31664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6805","nonce":31664},{"type":"v1","addr":"172.21.15.41:6806","nonce":31664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6812","nonce":31664},{"type":"v1","addr":"172.21.15.41:6814","nonce":31664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6808","nonce":31664},{"type":"v1","addr":"172.21.15.41:6810","nonce":31664}]},"public_addr":"172.21.15.41:6803/31664","cluster_addr":"172.21.15.41:6806/31664","heartbeat_back_addr":"172.21.15.41:6814/31664","heartbeat_front_addr":"172.21.15.41:6810/31664","state":["exists","up"]},{"osd":2,"uuid":"d6e49e26-6b08-4358-b1bb-c510ac35f34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6816","nonce":31665},{"type":"v1","addr":"172.21.15.41:6817","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6818","nonce":31665},{"type":"v1","addr":"172.21.15.41:6819","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6822","nonce":31665},{"type":"v1","addr":"172.21.15.41:6823","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6820","nonce":31665},{"type":"v1","addr":"172.21.15.41:6821","nonce":31665}]},"public_addr":"172.21.15.41:6817/31665","cluster_addr":"172.21.15.41:6819/31665","heartbeat_back_addr":"172.21.15.41:6823/31665","heartbeat_front_addr":"172.21.15.41:6821/31665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.944928+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.969957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.632428+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:42:52.031 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-09-06T12:42:50.023827+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2021-09-06T12:42:50.736568+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {}}] 2021-09-06T12:42:52.032 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-09-06T12:42:52.334 INFO:teuthology.orchestra.run.smithi041.stdout:pg_num: 8 2021-09-06T12:42:52.347 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get .mgr pg_num 2021-09-06T12:42:52.649 INFO:teuthology.orchestra.run.smithi041.stdout:pg_num: 1 2021-09-06T12:42:52.662 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-09-06T12:42:52.663 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-09-06T12:42:52.663 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-09-06T12:42:53.036 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:53.048 INFO:teuthology.orchestra.run.smithi041.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6824","nonce":31343},{"type":"v1","addr":"172.21.15.41:6825","nonce":31343}]},"active_addr":"172.21.15.41:6825/31343","active_change":"2021-09-06T12:42:46.689429+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.41:0","nonce":3396427798}]},{"addrvec":[{"type":"v2","addr":"172.21.15.41:0","nonce":1909123990}]},{"addrvec":[{"type":"v2","addr":"172.21.15.41:0","nonce":25595692}]}]}} 2021-09-06T12:42:53.051 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-09-06T12:42:53.051 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-09-06T12:42:53.052 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T12:42:53.460 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:53.460 INFO:teuthology.orchestra.run.smithi041.stdout:{"epoch":12,"fsid":"c2315c9e-41d7-497d-bdb5-f190256454f1","created":"2021-09-06T12:42:41.607676+0000","modified":"2021-09-06T12:42:52.681086+0000","last_up_change":"2021-09-06T12:42:49.575111+0000","last_in_change":"2021-09-06T12:42:42.559299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T12:42:50.023827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T12:42:50.736568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"81881c5c-becb-423b-89a0-1ecf30df70f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6800","nonce":31662},{"type":"v1","addr":"172.21.15.41:6802","nonce":31662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6804","nonce":31662},{"type":"v1","addr":"172.21.15.41:6807","nonce":31662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6813","nonce":31662},{"type":"v1","addr":"172.21.15.41:6815","nonce":31662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6809","nonce":31662},{"type":"v1","addr":"172.21.15.41:6811","nonce":31662}]},"public_addr":"172.21.15.41:6802/31662","cluster_addr":"172.21.15.41:6807/31662","heartbeat_back_addr":"172.21.15.41:6815/31662","heartbeat_front_addr":"172.21.15.41:6811/31662","state":["exists","up"]},{"osd":1,"uuid":"ba135a35-84bc-400f-89ec-a1d383f0a26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6801","nonce":31664},{"type":"v1","addr":"172.21.15.41:6803","nonce":31664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6805","nonce":31664},{"type":"v1","addr":"172.21.15.41:6806","nonce":31664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6812","nonce":31664},{"type":"v1","addr":"172.21.15.41:6814","nonce":31664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6808","nonce":31664},{"type":"v1","addr":"172.21.15.41:6810","nonce":31664}]},"public_addr":"172.21.15.41:6803/31664","cluster_addr":"172.21.15.41:6806/31664","heartbeat_back_addr":"172.21.15.41:6814/31664","heartbeat_front_addr":"172.21.15.41:6810/31664","state":["exists","up"]},{"osd":2,"uuid":"d6e49e26-6b08-4358-b1bb-c510ac35f34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6816","nonce":31665},{"type":"v1","addr":"172.21.15.41:6817","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6818","nonce":31665},{"type":"v1","addr":"172.21.15.41:6819","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6822","nonce":31665},{"type":"v1","addr":"172.21.15.41:6823","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6820","nonce":31665},{"type":"v1","addr":"172.21.15.41:6821","nonce":31665}]},"public_addr":"172.21.15.41:6817/31665","cluster_addr":"172.21.15.41:6819/31665","heartbeat_back_addr":"172.21.15.41:6823/31665","heartbeat_front_addr":"172.21.15.41:6821/31665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.944928+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.969957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.632428+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:42:53.475 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-09-06T12:42:53.476 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-09-06T12:42:53.780 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:53.780 INFO:teuthology.orchestra.run.smithi041.stdout:{"epoch":13,"fsid":"c2315c9e-41d7-497d-bdb5-f190256454f1","created":"2021-09-06T12:42:41.607676+0000","modified":"2021-09-06T12:42:53.686240+0000","last_up_change":"2021-09-06T12:42:49.575111+0000","last_in_change":"2021-09-06T12:42:42.559299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-09-06T12:42:50.023827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2021-09-06T12:42:50.736568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"81881c5c-becb-423b-89a0-1ecf30df70f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6800","nonce":31662},{"type":"v1","addr":"172.21.15.41:6802","nonce":31662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6804","nonce":31662},{"type":"v1","addr":"172.21.15.41:6807","nonce":31662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6813","nonce":31662},{"type":"v1","addr":"172.21.15.41:6815","nonce":31662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6809","nonce":31662},{"type":"v1","addr":"172.21.15.41:6811","nonce":31662}]},"public_addr":"172.21.15.41:6802/31662","cluster_addr":"172.21.15.41:6807/31662","heartbeat_back_addr":"172.21.15.41:6815/31662","heartbeat_front_addr":"172.21.15.41:6811/31662","state":["exists","up"]},{"osd":1,"uuid":"ba135a35-84bc-400f-89ec-a1d383f0a26f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6801","nonce":31664},{"type":"v1","addr":"172.21.15.41:6803","nonce":31664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6805","nonce":31664},{"type":"v1","addr":"172.21.15.41:6806","nonce":31664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6812","nonce":31664},{"type":"v1","addr":"172.21.15.41:6814","nonce":31664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6808","nonce":31664},{"type":"v1","addr":"172.21.15.41:6810","nonce":31664}]},"public_addr":"172.21.15.41:6803/31664","cluster_addr":"172.21.15.41:6806/31664","heartbeat_back_addr":"172.21.15.41:6814/31664","heartbeat_front_addr":"172.21.15.41:6810/31664","state":["exists","up"]},{"osd":2,"uuid":"d6e49e26-6b08-4358-b1bb-c510ac35f34d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6816","nonce":31665},{"type":"v1","addr":"172.21.15.41:6817","nonce":31665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6818","nonce":31665},{"type":"v1","addr":"172.21.15.41:6819","nonce":31665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6822","nonce":31665},{"type":"v1","addr":"172.21.15.41:6823","nonce":31665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.41:6820","nonce":31665},{"type":"v1","addr":"172.21.15.41:6821","nonce":31665}]},"public_addr":"172.21.15.41:6817/31665","cluster_addr":"172.21.15.41:6819/31665","heartbeat_back_addr":"172.21.15.41:6823/31665","heartbeat_front_addr":"172.21.15.41:6821/31665","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.944928+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.969957+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:42:46.632428+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:42:53.793 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-09-06T12:42:53.943 INFO:teuthology.orchestra.run.smithi041.stdout:38654705668 2021-09-06T12:42:53.944 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-09-06T12:42:54.095 INFO:teuthology.orchestra.run.smithi041.stdout:38654705667 2021-09-06T12:42:54.096 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-09-06T12:42:54.246 INFO:teuthology.orchestra.run.smithi041.stdout:34359738372 2021-09-06T12:42:54.247 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-06T12:42:54.550 INFO:teuthology.orchestra.run.smithi041.stdout:38654705667 2021-09-06T12:42:54.563 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705667 for osd.0 2021-09-06T12:42:55.565 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-09-06T12:42:55.883 INFO:teuthology.orchestra.run.smithi041.stdout:38654705668 2021-09-06T12:42:55.897 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705668 got 38654705668 for osd.0 2021-09-06T12:42:55.897 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-09-06T12:42:56.198 INFO:teuthology.orchestra.run.smithi041.stdout:38654705667 2021-09-06T12:42:56.212 INFO:tasks.ceph.ceph_manager.c1:need seq 38654705667 got 38654705667 for osd.1 2021-09-06T12:42:56.212 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-09-06T12:42:56.515 INFO:teuthology.orchestra.run.smithi041.stdout:34359738372 2021-09-06T12:42:56.528 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-09-06T12:42:56.528 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-09-06T12:42:56.528 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-06T12:42:56.830 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:56.832 INFO:teuthology.orchestra.run.smithi041.stderr:dumped all 2021-09-06T12:42:56.843 INFO:teuthology.orchestra.run.smithi041.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-09-06T12:42:56.694355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24860,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314547940,"statfs":{"total":322122547200,"available":322097090560,"internally_reserved":0,"allocated":1515520,"data_stored":1267602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000437"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926255+0000","last_change":"2021-09-06T12:42:51.692269+0000","last_active":"2021-09-06T12:42:52.926255+0000","last_peered":"2021-09-06T12:42:52.926255+0000","last_clean":"2021-09-06T12:42:52.926255+0000","last_became_active":"2021-09-06T12:42:51.691941+0000","last_became_peered":"2021-09-06T12:42:51.691941+0000","last_unstale":"2021-09-06T12:42:52.926255+0000","last_undegraded":"2021-09-06T12:42:52.926255+0000","last_fullsized":"2021-09-06T12:42:52.926255+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926307+0000","last_change":"2021-09-06T12:42:51.691493+0000","last_active":"2021-09-06T12:42:52.926307+0000","last_peered":"2021-09-06T12:42:52.926307+0000","last_clean":"2021-09-06T12:42:52.926307+0000","last_became_active":"2021-09-06T12:42:51.691181+0000","last_became_peered":"2021-09-06T12:42:51.691181+0000","last_unstale":"2021-09-06T12:42:52.926307+0000","last_undegraded":"2021-09-06T12:42:52.926307+0000","last_fullsized":"2021-09-06T12:42:52.926307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.694878+0000","last_change":"2021-09-06T12:42:51.691548+0000","last_active":"2021-09-06T12:42:51.694878+0000","last_peered":"2021-09-06T12:42:51.694878+0000","last_clean":"2021-09-06T12:42:51.694878+0000","last_became_active":"2021-09-06T12:42:51.691224+0000","last_became_peered":"2021-09-06T12:42:51.691224+0000","last_unstale":"2021-09-06T12:42:51.694878+0000","last_undegraded":"2021-09-06T12:42:51.694878+0000","last_fullsized":"2021-09-06T12:42:51.694878+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926344+0000","last_change":"2021-09-06T12:42:51.690370+0000","last_active":"2021-09-06T12:42:52.926344+0000","last_peered":"2021-09-06T12:42:52.926344+0000","last_clean":"2021-09-06T12:42:52.926344+0000","last_became_active":"2021-09-06T12:42:51.690040+0000","last_became_peered":"2021-09-06T12:42:51.690040+0000","last_unstale":"2021-09-06T12:42:52.926344+0000","last_undegraded":"2021-09-06T12:42:52.926344+0000","last_fullsized":"2021-09-06T12:42:52.926344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.694956+0000","last_change":"2021-09-06T12:42:51.692385+0000","last_active":"2021-09-06T12:42:51.694956+0000","last_peered":"2021-09-06T12:42:51.694956+0000","last_clean":"2021-09-06T12:42:51.694956+0000","last_became_active":"2021-09-06T12:42:51.692061+0000","last_became_peered":"2021-09-06T12:42:51.692061+0000","last_unstale":"2021-09-06T12:42:51.694956+0000","last_undegraded":"2021-09-06T12:42:51.694956+0000","last_fullsized":"2021-09-06T12:42:51.694956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":135,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:53.478511+0000","last_change":"2021-09-06T12:42:52.927907+0000","last_active":"2021-09-06T12:42:53.478511+0000","last_peered":"2021-09-06T12:42:53.478511+0000","last_clean":"2021-09-06T12:42:53.478511+0000","last_became_active":"2021-09-06T12:42:52.927155+0000","last_became_peered":"2021-09-06T12:42:52.927155+0000","last_unstale":"2021-09-06T12:42:53.478511+0000","last_undegraded":"2021-09-06T12:42:53.478511+0000","last_fullsized":"2021-09-06T12:42:53.478511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:51.675064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:51.675064+0000","last_clean_scrub_stamp":"2021-09-06T12:42:51.675064+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926245+0000","last_change":"2021-09-06T12:42:51.690561+0000","last_active":"2021-09-06T12:42:52.926245+0000","last_peered":"2021-09-06T12:42:52.926245+0000","last_clean":"2021-09-06T12:42:52.926245+0000","last_became_active":"2021-09-06T12:42:51.690247+0000","last_became_peered":"2021-09-06T12:42:51.690247+0000","last_unstale":"2021-09-06T12:42:52.926245+0000","last_undegraded":"2021-09-06T12:42:52.926245+0000","last_fullsized":"2021-09-06T12:42:52.926245+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926251+0000","last_change":"2021-09-06T12:42:51.691431+0000","last_active":"2021-09-06T12:42:52.926251+0000","last_peered":"2021-09-06T12:42:52.926251+0000","last_clean":"2021-09-06T12:42:52.926251+0000","last_became_active":"2021-09-06T12:42:51.691098+0000","last_became_peered":"2021-09-06T12:42:51.691098+0000","last_unstale":"2021-09-06T12:42:52.926251+0000","last_undegraded":"2021-09-06T12:42:52.926251+0000","last_fullsized":"2021-09-06T12:42:52.926251+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.697515+0000","last_change":"2021-09-06T12:42:51.694008+0000","last_active":"2021-09-06T12:42:51.697515+0000","last_peered":"2021-09-06T12:42:51.697515+0000","last_clean":"2021-09-06T12:42:51.697515+0000","last_became_active":"2021-09-06T12:42:51.693678+0000","last_became_peered":"2021-09-06T12:42:51.693678+0000","last_unstale":"2021-09-06T12:42:51.697515+0000","last_undegraded":"2021-09-06T12:42:51.697515+0000","last_fullsized":"2021-09-06T12:42:51.697515+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8572,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849028,"statfs":{"total":107374182400,"available":107365404672,"internally_reserved":0,"allocated":708608,"data_stored":621634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8688,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848912,"statfs":{"total":107374182400,"available":107365285888,"internally_reserved":0,"allocated":700416,"data_stored":618168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7600,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850000,"statfs":{"total":107374182400,"available":107366400000,"internally_reserved":0,"allocated":106496,"data_stored":27800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T12:42:56.845 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-09-06T12:42:57.141 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:57.143 INFO:teuthology.orchestra.run.smithi041.stderr:dumped all 2021-09-06T12:42:57.154 INFO:teuthology.orchestra.run.smithi041.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-09-06T12:42:56.694355+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24860,"kb_used_data":1480,"kb_used_omap":0,"kb_used_meta":23168,"kb_avail":314547940,"statfs":{"total":322122547200,"available":322097090560,"internally_reserved":0,"allocated":1515520,"data_stored":1267602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23724032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000437"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926255+0000","last_change":"2021-09-06T12:42:51.692269+0000","last_active":"2021-09-06T12:42:52.926255+0000","last_peered":"2021-09-06T12:42:52.926255+0000","last_clean":"2021-09-06T12:42:52.926255+0000","last_became_active":"2021-09-06T12:42:51.691941+0000","last_became_peered":"2021-09-06T12:42:51.691941+0000","last_unstale":"2021-09-06T12:42:52.926255+0000","last_undegraded":"2021-09-06T12:42:52.926255+0000","last_fullsized":"2021-09-06T12:42:52.926255+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926307+0000","last_change":"2021-09-06T12:42:51.691493+0000","last_active":"2021-09-06T12:42:52.926307+0000","last_peered":"2021-09-06T12:42:52.926307+0000","last_clean":"2021-09-06T12:42:52.926307+0000","last_became_active":"2021-09-06T12:42:51.691181+0000","last_became_peered":"2021-09-06T12:42:51.691181+0000","last_unstale":"2021-09-06T12:42:52.926307+0000","last_undegraded":"2021-09-06T12:42:52.926307+0000","last_fullsized":"2021-09-06T12:42:52.926307+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.694878+0000","last_change":"2021-09-06T12:42:51.691548+0000","last_active":"2021-09-06T12:42:51.694878+0000","last_peered":"2021-09-06T12:42:51.694878+0000","last_clean":"2021-09-06T12:42:51.694878+0000","last_became_active":"2021-09-06T12:42:51.691224+0000","last_became_peered":"2021-09-06T12:42:51.691224+0000","last_unstale":"2021-09-06T12:42:51.694878+0000","last_undegraded":"2021-09-06T12:42:51.694878+0000","last_fullsized":"2021-09-06T12:42:51.694878+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926344+0000","last_change":"2021-09-06T12:42:51.690370+0000","last_active":"2021-09-06T12:42:52.926344+0000","last_peered":"2021-09-06T12:42:52.926344+0000","last_clean":"2021-09-06T12:42:52.926344+0000","last_became_active":"2021-09-06T12:42:51.690040+0000","last_became_peered":"2021-09-06T12:42:51.690040+0000","last_unstale":"2021-09-06T12:42:52.926344+0000","last_undegraded":"2021-09-06T12:42:52.926344+0000","last_fullsized":"2021-09-06T12:42:52.926344+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.694956+0000","last_change":"2021-09-06T12:42:51.692385+0000","last_active":"2021-09-06T12:42:51.694956+0000","last_peered":"2021-09-06T12:42:51.694956+0000","last_clean":"2021-09-06T12:42:51.694956+0000","last_became_active":"2021-09-06T12:42:51.692061+0000","last_became_peered":"2021-09-06T12:42:51.692061+0000","last_unstale":"2021-09-06T12:42:51.694956+0000","last_undegraded":"2021-09-06T12:42:51.694956+0000","last_fullsized":"2021-09-06T12:42:51.694956+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"12'97","reported_seq":135,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:53.478511+0000","last_change":"2021-09-06T12:42:52.927907+0000","last_active":"2021-09-06T12:42:53.478511+0000","last_peered":"2021-09-06T12:42:53.478511+0000","last_clean":"2021-09-06T12:42:53.478511+0000","last_became_active":"2021-09-06T12:42:52.927155+0000","last_became_peered":"2021-09-06T12:42:52.927155+0000","last_unstale":"2021-09-06T12:42:53.478511+0000","last_undegraded":"2021-09-06T12:42:53.478511+0000","last_fullsized":"2021-09-06T12:42:53.478511+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:51.675064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:51.675064+0000","last_clean_scrub_stamp":"2021-09-06T12:42:51.675064+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926245+0000","last_change":"2021-09-06T12:42:51.690561+0000","last_active":"2021-09-06T12:42:52.926245+0000","last_peered":"2021-09-06T12:42:52.926245+0000","last_clean":"2021-09-06T12:42:52.926245+0000","last_became_active":"2021-09-06T12:42:51.690247+0000","last_became_peered":"2021-09-06T12:42:51.690247+0000","last_unstale":"2021-09-06T12:42:52.926245+0000","last_undegraded":"2021-09-06T12:42:52.926245+0000","last_fullsized":"2021-09-06T12:42:52.926245+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2021-09-06T12:42:52.926251+0000","last_change":"2021-09-06T12:42:51.691431+0000","last_active":"2021-09-06T12:42:52.926251+0000","last_peered":"2021-09-06T12:42:52.926251+0000","last_clean":"2021-09-06T12:42:52.926251+0000","last_became_active":"2021-09-06T12:42:51.691098+0000","last_became_peered":"2021-09-06T12:42:51.691098+0000","last_unstale":"2021-09-06T12:42:52.926251+0000","last_undegraded":"2021-09-06T12:42:52.926251+0000","last_fullsized":"2021-09-06T12:42:52.926251+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2021-09-06T12:42:51.697515+0000","last_change":"2021-09-06T12:42:51.694008+0000","last_active":"2021-09-06T12:42:51.697515+0000","last_peered":"2021-09-06T12:42:51.697515+0000","last_clean":"2021-09-06T12:42:51.697515+0000","last_became_active":"2021-09-06T12:42:51.693678+0000","last_became_peered":"2021-09-06T12:42:51.693678+0000","last_unstale":"2021-09-06T12:42:51.697515+0000","last_undegraded":"2021-09-06T12:42:51.697515+0000","last_fullsized":"2021-09-06T12:42:51.697515+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:42:50.669078+0000","last_clean_scrub_stamp":"2021-09-06T12:42:50.669078+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8572,"kb_used_data":692,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849028,"statfs":{"total":107374182400,"available":107365404672,"internally_reserved":0,"allocated":708608,"data_stored":621634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8688,"kb_used_data":684,"kb_used_omap":0,"kb_used_meta":7808,"kb_avail":104848912,"statfs":{"total":107374182400,"available":107365285888,"internally_reserved":0,"allocated":700416,"data_stored":618168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7995392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705669,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7600,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104850000,"statfs":{"total":107374182400,"available":107366400000,"internally_reserved":0,"allocated":106496,"data_stored":27800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T12:42:57.155 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-09-06T12:42:57.156 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-09-06T12:42:57.156 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-09-06T12:42:57.156 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-09-06T12:42:57.519 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:57.520 INFO:teuthology.orchestra.run.smithi041.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-06T12:42:57.532 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-09-06T12:42:57.533 INFO:teuthology.run_tasks:Running task ceph... 2021-09-06T12:42:57.543 INFO:tasks.ceph:Creating ceph cluster c2... 2021-09-06T12:42:57.543 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-09-06T12:42:57.544 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi/6376857', 'branch': 'wip-skoduri-lua', 'description': 'rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub supported-random-distro$/{rhel_8} tasks/test_multi}', 'email': None, 'first_in_suite': False, 'job_id': '6376857', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'soumyakoduri-2021-09-06_10:30:05-rgw-wip-skoduri-lua-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-skoduri-lua'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py']}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-skoduri-lua', 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73'}}, 'owner': 'scheduled_soumyakoduri@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-skoduri-lua', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'targets': {'smithi041.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtKEAsKjpnirpPJZlrVlDB6EUcJevmYskuRXgQf6uMRHU+EU2BE/X82jOB1CqMpIfr8BU2Ji7R/5NNG5JscS8FgZ06sUAoAo8g06h6Qf4JVI/JcRlmNOALGPSbdOQVranDh1SQn+Jhs02F/M+I1xdxwd63mXbfaGGEe4YpIuYozFVISgrXDNjEm3BOxUP9BUhWNyzVIFjB4uLvNKqP03Zpp1amOEJ38pXu9IlCC1RMFhSqw7eWQDih957x+Pl8LbUBtQlq4b+XRrkSCL+OwdboW3keev60KLitsQqYpvGlAdynHhRrC0tWiMOmTQa4fEyzf6fZFZVAC9AYG1X7TIiQg/cVokcFCG2iymZBE0LXbzGRulXWiDE4orhmlt818fI+SCJoJ7exNpUpe/7H7bdLDncrY7KVO4fbaK8IGp32jmHMzYdMJ6yRw9Io25LtFFY/Obz0EZW4UpV+ccEpwtuFKR0qjvAQjUdurnqHU7dB5Ujdi4o4Xv/sgr2AO5KEChc=', 'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDT1FWubehwPPALY5EhmmXs2T9Kf7zlSb7OkqXs4OzGfyd3xKEh38q10xEE4m6rgXG+GBpWHBJQvZUJSyBukInqY5N7EJ0/nzsyPQt/c7DcWSgvFuE4sA/ihpjg5ZikISU523gDbtc9AXqYX6wUouhh3LJ1ZI25LwHZQCkFON34Sii0I8wn58/9RtZY8AZ8C86UhbYZbVQvYwQJOYVPprf0rXSiQy4A1sZR4J/jm4oVU6VBtkzvzOBZNME011hmU5yWJ/RB6CPaf5nCO82HEBJorqfQ/QnJZeq3m5Jf87+mnOtitKe/ml8M6a0fcBq6XnGvPoD4o9bzRtrNg478pVxFOU+dbOI/YaV7zAtIWER2V5LRZgDzdmoFXBBxABf3h9PSiX3yPvT3PGbeNHAQctnGqLo1fYhBBKMt2mfxFkq/J5K+7GY8Z/Z1IlxE1qB479mOS+AiNxV6VwljlP5WIA2EkKjRAEvNjwr+xh12V2jzSNzH/IAYZ8R4/VfkmE6MUR8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '0eb1a7943dd70e2a0b3086ea680284137a187e73', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'timestamp': '2021-09-06_10:30:05', 'tube': 'smithi', 'user': 'soumyakoduri', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.30581'} 2021-09-06T12:42:57.545 DEBUG:teuthology.orchestra.run.smithi041:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-06T12:42:57.547 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-09-06T12:42:57.565 DEBUG:teuthology.orchestra.run.smithi041:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T12:42:57.602 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2021-09-06T12:42:57.635 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:57.636 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2021-09-06T12:42:57.692 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-09-06T12:42:57.693 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-09-06T12:42:57.749 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-09-06T12:42:57.749 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:57.750 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 114945 Links: 1 2021-09-06T12:42:57.750 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:57.750 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:57.750 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-09-06 12:41:46.716574449 +0000 2021-09-06T12:42:57.751 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-09-06 12:32:20.011759913 +0000 2021-09-06T12:42:57.751 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-09-06 12:32:20.011759913 +0000 2021-09-06T12:42:57.751 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-09-06T12:42:57.761 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-09-06T12:42:57.761 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-09-06T12:42:57.761 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000119371 s, 4.3 MB/s 2021-09-06T12:42:57.764 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-09-06T12:42:57.821 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-09-06T12:42:57.821 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:57.821 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 112981 Links: 1 2021-09-06T12:42:57.822 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:57.822 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:57.822 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-09-06 12:41:46.716574449 +0000 2021-09-06T12:42:57.822 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-09-06 12:32:20.356755655 +0000 2021-09-06T12:42:57.823 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-09-06 12:32:20.356755655 +0000 2021-09-06T12:42:57.823 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-09-06T12:42:57.832 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-09-06T12:42:57.832 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-09-06T12:42:57.832 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000111519 s, 4.6 MB/s 2021-09-06T12:42:57.835 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-09-06T12:42:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-09-06T12:42:57.892 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:57.892 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 113030 Links: 1 2021-09-06T12:42:57.892 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:57.893 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:57.893 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-09-06 12:41:46.716574449 +0000 2021-09-06T12:42:57.893 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-09-06 12:32:20.710751286 +0000 2021-09-06T12:42:57.893 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-09-06 12:32:20.710751286 +0000 2021-09-06T12:42:57.893 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-09-06T12:42:57.902 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-09-06T12:42:57.903 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-09-06T12:42:57.903 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000104872 s, 4.9 MB/s 2021-09-06T12:42:57.906 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-09-06T12:42:57.962 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-09-06T12:42:57.963 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-09-06T12:42:57.963 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 116051 Links: 1 2021-09-06T12:42:57.963 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-09-06T12:42:57.963 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2021-09-06T12:42:57.963 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-09-06 12:41:46.716574449 +0000 2021-09-06T12:42:57.964 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-09-06 12:32:21.076746769 +0000 2021-09-06T12:42:57.964 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-09-06 12:32:21.076746769 +0000 2021-09-06T12:42:57.964 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-09-06T12:42:57.973 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-09-06T12:42:57.973 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-09-06T12:42:57.974 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000107632 s, 4.8 MB/s 2021-09-06T12:42:57.976 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:57.977 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T12:42:57.977 INFO:tasks.ceph:Generating config... 2021-09-06T12:42:57.978 INFO:tasks.ceph:[client] debug rgw = 20 2021-09-06T12:42:57.979 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-09-06T12:42:57.979 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-09-06T12:42:57.979 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:57.980 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-09-06T12:42:57.980 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-09-06T12:42:57.981 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-09-06T12:42:57.981 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-09-06T12:42:57.981 INFO:tasks.ceph:[client] rgw sync meta inject err probability = 0.1 2021-09-06T12:42:57.981 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-09-06T12:42:57.982 INFO:tasks.ceph:[client] setgroup = ceph 2021-09-06T12:42:57.982 INFO:tasks.ceph:[client] setuser = ceph 2021-09-06T12:42:57.982 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-09-06T12:42:57.983 INFO:tasks.ceph:[mgr] debug ms = 1 2021-09-06T12:42:57.983 INFO:tasks.ceph:[mon] debug mon = 20 2021-09-06T12:42:57.983 INFO:tasks.ceph:[mon] debug ms = 1 2021-09-06T12:42:57.984 INFO:tasks.ceph:[mon] debug paxos = 20 2021-09-06T12:42:57.984 INFO:tasks.ceph:[osd] debug filestore = 20 2021-09-06T12:42:57.984 INFO:tasks.ceph:[osd] debug journal = 20 2021-09-06T12:42:57.984 INFO:tasks.ceph:[osd] debug ms = 20 2021-09-06T12:42:57.985 INFO:tasks.ceph:[osd] debug osd = 25 2021-09-06T12:42:57.985 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-09-06T12:42:57.985 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-09-06T12:42:57.986 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:57.986 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:57.986 INFO:tasks.ceph:Setting up c2.mon.a... 2021-09-06T12:42:57.987 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-09-06T12:42:58.075 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.keyring 2021-09-06T12:42:58.078 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-09-06T12:42:58.120 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T12:42:58.186 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.52')] 2021-09-06T12:42:58.186 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.52'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'rocksdb delete range threshold': 0}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync meta inject err probability': 0.1, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}} 2021-09-06T12:42:58.187 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:58.188 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T12:42:58.241 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.52 --print /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:42:58.318 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:42:58.319 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:42:58.320 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:42:58.320 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 8de2f231-5b80-41ad-90ee-e145a3077f13 2021-09-06T12:42:58.321 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = octopus 2021-09-06T12:42:58.321 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2021-09-06T12:42:58.321 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 8de2f231-5b80-41ad-90ee-e145a3077f13 2021-09-06T12:42:58.321 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2021-09-06T12:42:58.321062+0000 2021-09-06T12:42:58.322 INFO:teuthology.orchestra.run.smithi052.stdout:created 2021-09-06T12:42:58.321062+0000 2021-09-06T12:42:58.322 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 15 (octopus) 2021-09-06T12:42:58.322 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 1 2021-09-06T12:42:58.322 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2021-09-06T12:42:58.322 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-09-06T12:42:58.323 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-09-06T12:42:58.375 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 8de2f231-5b80-41ad-90ee-e145a3077f13... 2021-09-06T12:42:58.376 DEBUG:teuthology.orchestra.run.smithi041:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-06T12:42:58.379 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-09-06T12:42:58.429 INFO:teuthology.orchestra.run.smithi041.stdout:[global] 2021-09-06T12:42:58.429 INFO:teuthology.orchestra.run.smithi041.stdout: chdir = "" 2021-09-06T12:42:58.429 INFO:teuthology.orchestra.run.smithi041.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T12:42:58.429 INFO:teuthology.orchestra.run.smithi041.stdout: auth supported = cephx 2021-09-06T12:42:58.430 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.430 INFO:teuthology.orchestra.run.smithi041.stdout: filestore xattr use omap = true 2021-09-06T12:42:58.430 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.430 INFO:teuthology.orchestra.run.smithi041.stdout: mon clock drift allowed = 1.000 2021-09-06T12:42:58.431 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.431 INFO:teuthology.orchestra.run.smithi041.stdout: osd crush chooseleaf type = 0 2021-09-06T12:42:58.431 INFO:teuthology.orchestra.run.smithi041.stdout: auth debug = true 2021-09-06T12:42:58.431 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.432 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on old message = true 2021-09-06T12:42:58.432 INFO:teuthology.orchestra.run.smithi041.stdout: ms die on bug = true 2021-09-06T12:42:58.432 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.432 INFO:teuthology.orchestra.run.smithi041.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T12:42:58.432 INFO:teuthology.orchestra.run.smithi041.stdout: mon pg warn max object skew = 0 2021-09-06T12:42:58.433 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.433 INFO:teuthology.orchestra.run.smithi041.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T12:42:58.433 INFO:teuthology.orchestra.run.smithi041.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T12:42:58.433 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.433 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default size = 2 2021-09-06T12:42:58.434 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.434 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow primary affinity = true 2021-09-06T12:42:58.434 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd allow pg remap = true 2021-09-06T12:42:58.434 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on legacy crush tunables = false 2021-09-06T12:42:58.434 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on crush straw calc version zero = false 2021-09-06T12:42:58.435 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on no sortbitwise = false 2021-09-06T12:42:58.435 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on osd down out interval zero = false 2021-09-06T12:42:58.435 INFO:teuthology.orchestra.run.smithi041.stdout: mon warn on too few osds = false 2021-09-06T12:42:58.435 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T12:42:58.435 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T12:42:58.436 INFO:teuthology.orchestra.run.smithi041.stdout: mon_allow_pool_size_one = true 2021-09-06T12:42:58.436 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.436 INFO:teuthology.orchestra.run.smithi041.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T12:42:58.436 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.436 INFO:teuthology.orchestra.run.smithi041.stdout: osd default data pool replay window = 5 2021-09-06T12:42:58.437 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.437 INFO:teuthology.orchestra.run.smithi041.stdout: mon allow pool delete = true 2021-09-06T12:42:58.437 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.437 INFO:teuthology.orchestra.run.smithi041.stdout: mon cluster log file level = debug 2021-09-06T12:42:58.438 INFO:teuthology.orchestra.run.smithi041.stdout: debug asserts on shutdown = true 2021-09-06T12:42:58.438 INFO:teuthology.orchestra.run.smithi041.stdout: mon health detail to clog = false 2021-09-06T12:42:58.438 INFO:teuthology.orchestra.run.smithi041.stdout: mon host = 172.21.15.52 2021-09-06T12:42:58.438 INFO:teuthology.orchestra.run.smithi041.stdout: fsid = 8de2f231-5b80-41ad-90ee-e145a3077f13 2021-09-06T12:42:58.438 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.439 INFO:teuthology.orchestra.run.smithi041.stdout:[osd] 2021-09-06T12:42:58.439 INFO:teuthology.orchestra.run.smithi041.stdout: osd journal size = 100 2021-09-06T12:42:58.439 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.439 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub load threshold = 5.0 2021-09-06T12:42:58.439 INFO:teuthology.orchestra.run.smithi041.stdout: osd scrub max interval = 600 2021-09-06T12:42:58.440 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.440 INFO:teuthology.orchestra.run.smithi041.stdout: osd recover clone overlap = true 2021-09-06T12:42:58.440 INFO:teuthology.orchestra.run.smithi041.stdout: osd recovery max chunk = 1048576 2021-09-06T12:42:58.440 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.441 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug shutdown = true 2021-09-06T12:42:58.441 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug op order = true 2021-09-06T12:42:58.441 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug verify stray on activate = true 2021-09-06T12:42:58.441 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.441 INFO:teuthology.orchestra.run.smithi041.stdout: osd open classes on start = true 2021-09-06T12:42:58.442 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug pg log writeout = true 2021-09-06T12:42:58.442 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.442 INFO:teuthology.orchestra.run.smithi041.stdout: osd deep scrub update digest min age = 30 2021-09-06T12:42:58.442 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.442 INFO:teuthology.orchestra.run.smithi041.stdout: osd map max advance = 10 2021-09-06T12:42:58.443 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.443 INFO:teuthology.orchestra.run.smithi041.stdout: journal zero on create = true 2021-09-06T12:42:58.443 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.443 INFO:teuthology.orchestra.run.smithi041.stdout: filestore ondisk finisher threads = 3 2021-09-06T12:42:58.444 INFO:teuthology.orchestra.run.smithi041.stdout: filestore apply finisher threads = 3 2021-09-06T12:42:58.444 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.444 INFO:teuthology.orchestra.run.smithi041.stdout: bdev debug aio = true 2021-09-06T12:42:58.444 INFO:teuthology.orchestra.run.smithi041.stdout: osd debug misdirected ops = true 2021-09-06T12:42:58.444 INFO:teuthology.orchestra.run.smithi041.stdout: debug filestore = 20 2021-09-06T12:42:58.445 INFO:teuthology.orchestra.run.smithi041.stdout: debug journal = 20 2021-09-06T12:42:58.445 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 20 2021-09-06T12:42:58.445 INFO:teuthology.orchestra.run.smithi041.stdout: debug osd = 25 2021-09-06T12:42:58.445 INFO:teuthology.orchestra.run.smithi041.stdout: rocksdb delete range threshold = 0 2021-09-06T12:42:58.445 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.446 INFO:teuthology.orchestra.run.smithi041.stdout:[mgr] 2021-09-06T12:42:58.446 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-09-06T12:42:58.446 INFO:teuthology.orchestra.run.smithi041.stdout: debug mgr = 20 2021-09-06T12:42:58.446 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-09-06T12:42:58.447 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-09-06T12:42:58.447 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:58.447 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:58.447 INFO:teuthology.orchestra.run.smithi041.stdout: mgr/telemetry/nag = false 2021-09-06T12:42:58.447 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.448 INFO:teuthology.orchestra.run.smithi041.stdout:[mon] 2021-09-06T12:42:58.448 INFO:teuthology.orchestra.run.smithi041.stdout: debug ms = 1 2021-09-06T12:42:58.448 INFO:teuthology.orchestra.run.smithi041.stdout: debug mon = 20 2021-09-06T12:42:58.448 INFO:teuthology.orchestra.run.smithi041.stdout: debug paxos = 20 2021-09-06T12:42:58.448 INFO:teuthology.orchestra.run.smithi041.stdout: debug auth = 20 2021-09-06T12:42:58.449 INFO:teuthology.orchestra.run.smithi041.stdout: mon data avail warn = 5 2021-09-06T12:42:58.449 INFO:teuthology.orchestra.run.smithi041.stdout: mon mgr mkfs grace = 240 2021-09-06T12:42:58.449 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:58.449 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd reporter subtree level = osd 2021-09-06T12:42:58.450 INFO:teuthology.orchestra.run.smithi041.stdout: mon osd prime pg temp = true 2021-09-06T12:42:58.450 INFO:teuthology.orchestra.run.smithi041.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:58.450 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.450 INFO:teuthology.orchestra.run.smithi041.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T12:42:58.450 INFO:teuthology.orchestra.run.smithi041.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T12:42:58.451 INFO:teuthology.orchestra.run.smithi041.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T12:42:58.451 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.451 INFO:teuthology.orchestra.run.smithi041.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T12:42:58.451 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T12:42:58.451 INFO:teuthology.orchestra.run.smithi041.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T12:42:58.452 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:42:58.452 INFO:teuthology.orchestra.run.smithi041.stdout:[client] 2021-09-06T12:42:58.452 INFO:teuthology.orchestra.run.smithi041.stdout: rgw cache enabled = true 2021-09-06T12:42:58.452 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable ops log = true 2021-09-06T12:42:58.452 INFO:teuthology.orchestra.run.smithi041.stdout: rgw enable usage log = true 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: debug rgw = 20 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt require ssl = False 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt s3 kms backend = testing 2021-09-06T12:42:58.453 INFO:teuthology.orchestra.run.smithi041.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:58.454 INFO:teuthology.orchestra.run.smithi041.stdout: rgw curl low speed time = 300 2021-09-06T12:42:58.454 INFO:teuthology.orchestra.run.smithi041.stdout: rgw data log num shards = 4 2021-09-06T12:42:58.454 INFO:teuthology.orchestra.run.smithi041.stdout: rgw md log max shards = 4 2021-09-06T12:42:58.454 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync log trim interval = 0 2021-09-06T12:42:58.455 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T12:42:58.455 INFO:teuthology.orchestra.run.smithi041.stdout: rgw sync obj etag verify = True 2021-09-06T12:42:58.455 INFO:teuthology.orchestra.run.smithi041.stdout: setgroup = ceph 2021-09-06T12:42:58.455 INFO:teuthology.orchestra.run.smithi041.stdout: setuser = ceph 2021-09-06T12:42:58.455 INFO:teuthology.orchestra.run.smithi041.stdout:[mon.a] 2021-09-06T12:42:58.456 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.0] 2021-09-06T12:42:58.456 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T12:42:58.456 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.1] 2021-09-06T12:42:58.456 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:58.456 INFO:teuthology.orchestra.run.smithi041.stdout:[osd.2] 2021-09-06T12:42:58.457 INFO:teuthology.orchestra.run.smithi041.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:58.459 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2021-09-06T12:42:58.460 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2021-09-06T12:42:58.460 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-09-06T12:42:58.460 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2021-09-06T12:42:58.460 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.460 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2021-09-06T12:42:58.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.461 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2021-09-06T12:42:58.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.461 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2021-09-06T12:42:58.462 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2021-09-06T12:42:58.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.462 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2021-09-06T12:42:58.462 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2021-09-06T12:42:58.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.463 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2021-09-06T12:42:58.463 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2021-09-06T12:42:58.463 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.463 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2021-09-06T12:42:58.464 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2021-09-06T12:42:58.464 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.464 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2021-09-06T12:42:58.464 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.464 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2021-09-06T12:42:58.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2021-09-06T12:42:58.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2021-09-06T12:42:58.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2021-09-06T12:42:58.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2021-09-06T12:42:58.466 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2021-09-06T12:42:58.466 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2021-09-06T12:42:58.466 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-09-06T12:42:58.466 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2021-09-06T12:42:58.466 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2021-09-06T12:42:58.467 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.467 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-09-06T12:42:58.467 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.467 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2021-09-06T12:42:58.467 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2021-09-06T12:42:58.468 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2021-09-06T12:42:58.468 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2021-09-06T12:42:58.469 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2021-09-06T12:42:58.469 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.52 2021-09-06T12:42:58.469 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 8de2f231-5b80-41ad-90ee-e145a3077f13 2021-09-06T12:42:58.469 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.469 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2021-09-06T12:42:58.470 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2021-09-06T12:42:58.470 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.470 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2021-09-06T12:42:58.470 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2021-09-06T12:42:58.470 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.471 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2021-09-06T12:42:58.471 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2021-09-06T12:42:58.471 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.471 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2021-09-06T12:42:58.472 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2021-09-06T12:42:58.472 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2021-09-06T12:42:58.472 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.472 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2021-09-06T12:42:58.472 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2021-09-06T12:42:58.473 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.473 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2021-09-06T12:42:58.473 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.473 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2021-09-06T12:42:58.473 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.474 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2021-09-06T12:42:58.474 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.474 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2021-09-06T12:42:58.474 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2021-09-06T12:42:58.475 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.475 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2021-09-06T12:42:58.475 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2021-09-06T12:42:58.475 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 20 2021-09-06T12:42:58.476 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 20 2021-09-06T12:42:58.476 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 20 2021-09-06T12:42:58.476 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 25 2021-09-06T12:42:58.476 INFO:teuthology.orchestra.run.smithi052.stdout: rocksdb delete range threshold = 0 2021-09-06T12:42:58.476 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.477 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2021-09-06T12:42:58.477 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-09-06T12:42:58.477 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2021-09-06T12:42:58.477 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-09-06T12:42:58.477 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-09-06T12:42:58.478 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:58.478 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:58.478 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2021-09-06T12:42:58.478 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.479 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2021-09-06T12:42:58.479 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-09-06T12:42:58.479 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-09-06T12:42:58.479 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2021-09-06T12:42:58.479 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-09-06T12:42:58.480 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2021-09-06T12:42:58.480 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2021-09-06T12:42:58.480 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-09-06T12:42:58.480 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2021-09-06T12:42:58.480 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2021-09-06T12:42:58.481 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-09-06T12:42:58.481 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.481 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2021-09-06T12:42:58.481 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2021-09-06T12:42:58.482 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2021-09-06T12:42:58.482 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.482 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2021-09-06T12:42:58.482 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2021-09-06T12:42:58.482 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2021-09-06T12:42:58.483 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:42:58.483 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2021-09-06T12:42:58.483 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2021-09-06T12:42:58.483 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2021-09-06T12:42:58.483 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2021-09-06T12:42:58.484 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-09-06T12:42:58.484 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-09-06T12:42:58.484 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2021-09-06T12:42:58.484 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2021-09-06T12:42:58.484 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2021-09-06T12:42:58.485 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-09-06T12:42:58.485 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2021-09-06T12:42:58.485 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2021-09-06T12:42:58.485 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2021-09-06T12:42:58.486 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2021-09-06T12:42:58.486 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync meta inject err probability = 0.1 2021-09-06T12:42:58.486 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2021-09-06T12:42:58.486 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2021-09-06T12:42:58.486 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2021-09-06T12:42:58.487 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2021-09-06T12:42:58.487 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2021-09-06T12:42:58.487 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10 2021-09-06T12:42:58.487 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2021-09-06T12:42:58.487 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 1000 2021-09-06T12:42:58.488 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2021-09-06T12:42:58.488 INFO:teuthology.orchestra.run.smithi052.stdout: osd_max_omap_entries_per_request = 10000 2021-09-06T12:42:58.489 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-09-06T12:42:58.489 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-09-06T12:42:58.540 INFO:tasks.ceph:Copying monmap to all nodes... 2021-09-06T12:42:58.541 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:58.541 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-09-06T12:42:58.556 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:58.556 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-09-06T12:42:58.611 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi041.front.sepia.ceph.com 2021-09-06T12:42:58.611 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:58.611 DEBUG:teuthology.orchestra.run.smithi041:> sudo dd of=/etc/ceph/c2.keyring 2021-09-06T12:42:58.612 DEBUG:teuthology.orchestra.run.smithi041:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T12:42:58.653 DEBUG:teuthology.orchestra.run.smithi041:> set -ex 2021-09-06T12:42:58.654 DEBUG:teuthology.orchestra.run.smithi041:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-06T12:42:58.711 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:42:58.711 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:58.711 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/c2.keyring 2021-09-06T12:42:58.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-09-06T12:42:58.747 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:42:58.747 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-09-06T12:42:58.801 INFO:tasks.ceph:Setting up mon nodes... 2021-09-06T12:42:58.802 INFO:tasks.ceph:Setting up mgr nodes... 2021-09-06T12:42:58.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-09-06T12:42:58.889 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-09-06T12:42:58.891 INFO:tasks.ceph:Setting up mds nodes... 2021-09-06T12:42:58.892 INFO:tasks.ceph_client:Setting up client nodes... 2021-09-06T12:42:58.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-09-06T12:42:58.930 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.client.0.keyring 2021-09-06T12:42:58.942 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-09-06T12:42:59.020 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c2.client.1.keyring 2021-09-06T12:42:59.032 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-09-06T12:42:59.032 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi041.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-09-06T12:42:59.033 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-09-06T12:42:59.096 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:59.096 INFO:tasks.ceph:role: c2.osd.0 2021-09-06T12:42:59.097 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:42:59.097 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-09-06T12:42:59.160 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:59.160 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:59.160 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:59.160 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 2021-09-06T12:42:59.161 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:59.161 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:59.161 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:59.161 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:59.162 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:59.162 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:59.183 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2021-09-06T12:42:59.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-09-06T12:42:59.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-09-06T12:42:59.263 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-09-06T12:42:59.333 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-09-06T12:42:59.398 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:59.398 INFO:tasks.ceph:role: c2.osd.1 2021-09-06T12:42:59.398 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:42:59.399 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-09-06T12:42:59.462 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:59.462 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:59.463 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:59.463 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 2021-09-06T12:42:59.463 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:59.463 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:59.463 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:59.464 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:59.464 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:59.464 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:59.485 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2021-09-06T12:42:59.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-09-06T12:42:59.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-09-06T12:42:59.564 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-09-06T12:42:59.632 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-09-06T12:42:59.697 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-09-06T12:42:59.697 INFO:tasks.ceph:role: c2.osd.2 2021-09-06T12:42:59.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2021-09-06T12:42:59.698 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-09-06T12:42:59.761 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-09-06T12:42:59.761 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-09-06T12:42:59.761 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-09-06T12:42:59.761 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 2021-09-06T12:42:59.762 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-09-06T12:42:59.762 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-09-06T12:42:59.762 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-09-06T12:42:59.762 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-09-06T12:42:59.762 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-09-06T12:42:59.763 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-09-06T12:42:59.783 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2021-09-06T12:42:59.785 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-09-06T12:42:59.785 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-09-06T12:42:59.862 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-09-06T12:42:59.930 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:43:00.012 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.011+0000 7f7679fc00c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-09-06T12:43:00.012 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.012+0000 7f7679fc00c0 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-09-06T12:43:00.012 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.012+0000 7f7679fc00c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T12:43:00.013 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.012+0000 7f7679fc00c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T12:43:00.013 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.012+0000 7f7679fc00c0 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-09-06T12:43:00.013 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:00.012+0000 7f7679fc00c0 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-09-06T12:43:02.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-09-06T12:43:02.166 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:43:02.253 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-09-06T12:43:02.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-09-06T12:43:02.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T12:43:02.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T12:43:02.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-09-06T12:43:02.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:02.253+0000 7fc7ef6020c0 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-09-06T12:43:04.334 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-09-06T12:43:04.364 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:43:04.452 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.452+0000 7f3a773490c0 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-09-06T12:43:04.453 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.452+0000 7f3a773490c0 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-09-06T12:43:04.453 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.452+0000 7f3a773490c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T12:43:04.453 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.452+0000 7f3a773490c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T12:43:04.453 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.452+0000 7f3a773490c0 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-09-06T12:43:04.453 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:04.453+0000 7f3a773490c0 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-09-06T12:43:06.533 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-09-06T12:43:06.562 INFO:tasks.ceph:Reading keys from all nodes... 2021-09-06T12:43:06.563 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.563 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-09-06T12:43:06.630 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.630 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-09-06T12:43:06.697 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.697 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-09-06T12:43:06.764 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.765 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-09-06T12:43:06.831 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.831 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-09-06T12:43:06.887 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:06.888 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-09-06T12:43:06.943 INFO:tasks.ceph:Adding keys to all mons... 2021-09-06T12:43:06.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/c2.keyring 2021-09-06T12:43:07.008 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2021-09-06T12:43:07.008 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDSDDZhQTEiNRAADDVGgjIjsapiCBWNcxQwOA== 2021-09-06T12:43:07.009 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2021-09-06T12:43:07.009 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDUDDZhUKLLABAAHssJBTUxNc8FukUHKv5YvA== 2021-09-06T12:43:07.009 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2021-09-06T12:43:07.009 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDWDDZhKBIyDxAAYFb0H3HVvJ4ZROQ4O/kJNw== 2021-09-06T12:43:07.010 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2021-09-06T12:43:07.010 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDYDDZhKPwSGxAA8j0wOn+QxAwPedkiywj8zQ== 2021-09-06T12:43:07.010 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2021-09-06T12:43:07.010 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDSDDZhZm+VNxAAmBKfD7rJPFSmhafYEePnGw== 2021-09-06T12:43:07.010 INFO:teuthology.orchestra.run.smithi052.stdout:[client.1] 2021-09-06T12:43:07.011 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQDTDDZhO0JTARAAKkxiQl725OqV4oc+EGn25Q== 2021-09-06T12:43:07.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-09-06T12:43:07.101 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:43:07.149 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:43:07.196 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-09-06T12:43:07.243 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T12:43:07.294 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-09-06T12:43:07.344 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-09-06T12:43:07.344 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-09-06T12:43:07.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-09-06T12:43:07.461 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-09-06T12:43:07.486 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-09-06T12:43:07.540 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-09-06T12:43:07.541 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-09-06T12:43:07.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-09-06T12:43:07.582 INFO:tasks.ceph.c2.mon.a:Started 2021-09-06T12:43:07.583 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-09-06T12:43:07.583 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-09-06T12:43:07.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-09-06T12:43:07.586 INFO:tasks.ceph.c2.mgr.x:Started 2021-09-06T12:43:07.586 INFO:tasks.ceph:Setting crush tunables to default 2021-09-06T12:43:07.587 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd crush tunables default 2021-09-06T12:43:07.787 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2021-09-06T12:43:07.798 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-09-06T12:43:07.799 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:07.799 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-09-06T12:43:07.823 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:07.824 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-09-06T12:43:07.887 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-09-06T12:43:07.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-09-06T12:43:07.952 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new 73861d13-b3af-4fc8-b5d5-2911da2b6f9c 0 2021-09-06T12:43:08.201 INFO:teuthology.orchestra.run.smithi052.stdout:0 2021-09-06T12:43:08.214 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new de1d9fa3-a1f7-43f9-9bff-3b0251a52399 1 2021-09-06T12:43:08.431 INFO:teuthology.orchestra.run.smithi052.stdout:1 2021-09-06T12:43:08.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd new 8b47c2ac-d408-4381-abaf-413e33f6d6c2 2 2021-09-06T12:43:08.652 INFO:teuthology.orchestra.run.smithi052.stdout:2 2021-09-06T12:43:08.664 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-09-06T12:43:08.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-09-06T12:43:08.667 INFO:tasks.ceph.c2.osd.0:Started 2021-09-06T12:43:08.668 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-09-06T12:43:08.668 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-09-06T12:43:08.670 INFO:tasks.ceph.c2.osd.1:Started 2021-09-06T12:43:08.671 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-09-06T12:43:08.671 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-09-06T12:43:08.673 INFO:tasks.ceph.c2.osd.2:Started 2021-09-06T12:43:08.674 INFO:tasks.ceph:Waiting for OSDs to come up 2021-09-06T12:43:08.976 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:09.162 INFO:teuthology.misc.health.smithi052.stdout: 2021-09-06T12:43:09.163 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":5,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:08.652717+0000","last_up_change":"0.000000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:09.173 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-09-06T12:43:09.264 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2021-09-06T12:43:09.264+0000 7f074d55a0c0 -1 Falling back to public interface 2021-09-06T12:43:09.267 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2021-09-06T12:43:09.267+0000 7f8b419690c0 -1 Falling back to public interface 2021-09-06T12:43:09.269 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2021-09-06T12:43:09.269+0000 7f324dfbe0c0 -1 Falling back to public interface 2021-09-06T12:43:11.744 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2021-09-06T12:43:11.744+0000 7f074d55a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-09-06T12:43:11.764 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2021-09-06T12:43:11.764+0000 7f324dfbe0c0 -1 osd.2 0 log_to_monitors {default=true} 2021-09-06T12:43:11.995 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2021-09-06T12:43:11.996+0000 7f8b419690c0 -1 osd.1 0 log_to_monitors {default=true} 2021-09-06T12:43:13.736 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2021-09-06T12:43:13.736+0000 7f324497c700 -1 osd.2 0 waiting for initial osdmap 2021-09-06T12:43:13.738 INFO:tasks.ceph.c2.osd.2.smithi052.stderr:2021-09-06T12:43:13.738+0000 7f323d30f700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:43:13.762 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2021-09-06T12:43:13.762+0000 7f0743f18700 -1 osd.0 0 waiting for initial osdmap 2021-09-06T12:43:13.763 INFO:tasks.ceph.c2.osd.0.smithi052.stderr:2021-09-06T12:43:13.764+0000 7f073c8ab700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:43:14.879 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2021-09-06T12:43:14.879+0000 7f8b38327700 -1 osd.1 0 waiting for initial osdmap 2021-09-06T12:43:14.881 INFO:tasks.ceph.c2.osd.1.smithi052.stderr:2021-09-06T12:43:14.881+0000 7f8b30cba700 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2021-09-06T12:43:15.476 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:15.781 INFO:teuthology.misc.health.smithi052.stdout: 2021-09-06T12:43:15.782 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":9,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:14.806267+0000","last_up_change":"2021-09-06T12:43:13.803383+0000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":31600},{"type":"v1","addr":"172.21.15.52:6801","nonce":31600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":31600},{"type":"v1","addr":"172.21.15.52:6803","nonce":31600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":31600},{"type":"v1","addr":"172.21.15.52:6807","nonce":31600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":31600},{"type":"v1","addr":"172.21.15.52:6805","nonce":31600}]},"public_addr":"172.21.15.52:6801/31600","cluster_addr":"172.21.15.52:6803/31600","heartbeat_back_addr":"172.21.15.52:6807/31600","heartbeat_front_addr":"172.21.15.52:6805/31600","state":["exists","up"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":31602},{"type":"v1","addr":"172.21.15.52:6816","nonce":31602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":31602},{"type":"v1","addr":"172.21.15.52:6819","nonce":31602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":31602},{"type":"v1","addr":"172.21.15.52:6823","nonce":31602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":31602},{"type":"v1","addr":"172.21.15.52:6821","nonce":31602}]},"public_addr":"172.21.15.52:6816/31602","cluster_addr":"172.21.15.52:6819/31602","heartbeat_back_addr":"172.21.15.52:6823/31602","heartbeat_front_addr":"172.21.15.52:6821/31602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:15.793 DEBUG:teuthology.misc:2 of 3 OSDs are up 2021-09-06T12:43:22.101 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:22.408 INFO:teuthology.misc.health.smithi052.stdout: 2021-09-06T12:43:22.409 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":12,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:18.667359+0000","last_up_change":"2021-09-06T12:43:15.808870+0000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-06T12:43:14.841634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":31600},{"type":"v1","addr":"172.21.15.52:6801","nonce":31600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":31600},{"type":"v1","addr":"172.21.15.52:6803","nonce":31600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":31600},{"type":"v1","addr":"172.21.15.52:6807","nonce":31600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":31600},{"type":"v1","addr":"172.21.15.52:6805","nonce":31600}]},"public_addr":"172.21.15.52:6801/31600","cluster_addr":"172.21.15.52:6803/31600","heartbeat_back_addr":"172.21.15.52:6807/31600","heartbeat_front_addr":"172.21.15.52:6805/31600","state":["exists","up"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":31601},{"type":"v1","addr":"172.21.15.52:6809","nonce":31601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":31601},{"type":"v1","addr":"172.21.15.52:6811","nonce":31601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6815","nonce":31601},{"type":"v1","addr":"172.21.15.52:6817","nonce":31601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":31601},{"type":"v1","addr":"172.21.15.52:6813","nonce":31601}]},"public_addr":"172.21.15.52:6809/31601","cluster_addr":"172.21.15.52:6811/31601","heartbeat_back_addr":"172.21.15.52:6817/31601","heartbeat_front_addr":"172.21.15.52:6813/31601","state":["exists","up"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":31602},{"type":"v1","addr":"172.21.15.52:6816","nonce":31602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":31602},{"type":"v1","addr":"172.21.15.52:6819","nonce":31602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":31602},{"type":"v1","addr":"172.21.15.52:6823","nonce":31602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":31602},{"type":"v1","addr":"172.21.15.52:6821","nonce":31602}]},"public_addr":"172.21.15.52:6816/31602","cluster_addr":"172.21.15.52:6819/31602","heartbeat_back_addr":"172.21.15.52:6823/31602","heartbeat_front_addr":"172.21.15.52:6821/31602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.991216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:22.420 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-09-06T12:43:22.421 INFO:tasks.ceph:Creating RBD pool 2021-09-06T12:43:22.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd pool create rbd 8 2021-09-06T12:43:23.682 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'rbd' created 2021-09-06T12:43:23.695 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-09-06T12:43:24.686 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rbd' on pool 'rbd' 2021-09-06T12:43:24.699 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-09-06T12:43:24.700 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:24.701 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-09-06T12:43:25.008 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:25.008 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":14,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:24.676047+0000","last_up_change":"2021-09-06T12:43:15.808870+0000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-06T12:43:14.841634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-06T12:43:22.751582+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":31600},{"type":"v1","addr":"172.21.15.52:6801","nonce":31600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":31600},{"type":"v1","addr":"172.21.15.52:6803","nonce":31600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":31600},{"type":"v1","addr":"172.21.15.52:6807","nonce":31600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":31600},{"type":"v1","addr":"172.21.15.52:6805","nonce":31600}]},"public_addr":"172.21.15.52:6801/31600","cluster_addr":"172.21.15.52:6803/31600","heartbeat_back_addr":"172.21.15.52:6807/31600","heartbeat_front_addr":"172.21.15.52:6805/31600","state":["exists","up"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":31601},{"type":"v1","addr":"172.21.15.52:6809","nonce":31601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":31601},{"type":"v1","addr":"172.21.15.52:6811","nonce":31601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6815","nonce":31601},{"type":"v1","addr":"172.21.15.52:6817","nonce":31601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":31601},{"type":"v1","addr":"172.21.15.52:6813","nonce":31601}]},"public_addr":"172.21.15.52:6809/31601","cluster_addr":"172.21.15.52:6811/31601","heartbeat_back_addr":"172.21.15.52:6817/31601","heartbeat_front_addr":"172.21.15.52:6813/31601","state":["exists","up"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":31602},{"type":"v1","addr":"172.21.15.52:6816","nonce":31602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":31602},{"type":"v1","addr":"172.21.15.52:6819","nonce":31602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":31602},{"type":"v1","addr":"172.21.15.52:6823","nonce":31602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":31602},{"type":"v1","addr":"172.21.15.52:6821","nonce":31602}]},"public_addr":"172.21.15.52:6816/31602","cluster_addr":"172.21.15.52:6819/31602","heartbeat_back_addr":"172.21.15.52:6823/31602","heartbeat_front_addr":"172.21.15.52:6821/31602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.991216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:25.021 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2021-09-06T12:43:14.841634+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-09-06T12:43:22.751582+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-09-06T12:43:25.021 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get .mgr pg_num 2021-09-06T12:43:25.326 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2021-09-06T12:43:25.339 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-09-06T12:43:25.643 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 8 2021-09-06T12:43:25.656 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-09-06T12:43:25.656 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-09-06T12:43:25.657 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-09-06T12:43:26.009 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:26.021 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":31281},{"type":"v1","addr":"172.21.15.52:6825","nonce":31281}]},"active_addr":"172.21.15.52:6825/31281","active_change":"2021-09-06T12:43:12.790432+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Registry to which we should normalize unqualified image names","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"autoscale_profile":{"name":"autoscale_profile","type":"str","level":"advanced","flags":1,"default_value":"scale-up","min":"","max":"","enum_allowed":[],"desc":"pg_autoscale profiler","long_desc":"Determines the behavior of the autoscaler algorithm `scale-up` means that it starts out with minmum pgs and scales up when there is pressure, `scale-down` means starts out with full pgs and scales down when there is pressure ","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share perf counter metrics summed across the whole cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":4088449541}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":3802424769}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1860808904}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":4233462793}]}]}} 2021-09-06T12:43:26.023 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-09-06T12:43:26.023 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-09-06T12:43:26.024 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:26.326 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:26.326 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":15,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:25.680178+0000","last_up_change":"2021-09-06T12:43:15.808870+0000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-06T12:43:14.841634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-06T12:43:22.751582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":31600},{"type":"v1","addr":"172.21.15.52:6801","nonce":31600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":31600},{"type":"v1","addr":"172.21.15.52:6803","nonce":31600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":31600},{"type":"v1","addr":"172.21.15.52:6807","nonce":31600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":31600},{"type":"v1","addr":"172.21.15.52:6805","nonce":31600}]},"public_addr":"172.21.15.52:6801/31600","cluster_addr":"172.21.15.52:6803/31600","heartbeat_back_addr":"172.21.15.52:6807/31600","heartbeat_front_addr":"172.21.15.52:6805/31600","state":["exists","up"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":31601},{"type":"v1","addr":"172.21.15.52:6809","nonce":31601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":31601},{"type":"v1","addr":"172.21.15.52:6811","nonce":31601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6815","nonce":31601},{"type":"v1","addr":"172.21.15.52:6817","nonce":31601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":31601},{"type":"v1","addr":"172.21.15.52:6813","nonce":31601}]},"public_addr":"172.21.15.52:6809/31601","cluster_addr":"172.21.15.52:6811/31601","heartbeat_back_addr":"172.21.15.52:6817/31601","heartbeat_front_addr":"172.21.15.52:6813/31601","state":["exists","up"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":31602},{"type":"v1","addr":"172.21.15.52:6816","nonce":31602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":31602},{"type":"v1","addr":"172.21.15.52:6819","nonce":31602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":31602},{"type":"v1","addr":"172.21.15.52:6823","nonce":31602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":31602},{"type":"v1","addr":"172.21.15.52:6821","nonce":31602}]},"public_addr":"172.21.15.52:6816/31602","cluster_addr":"172.21.15.52:6819/31602","heartbeat_back_addr":"172.21.15.52:6823/31602","heartbeat_front_addr":"172.21.15.52:6821/31602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.991216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:26.340 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-09-06T12:43:26.340 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-09-06T12:43:26.639 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:26.639 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":15,"fsid":"8de2f231-5b80-41ad-90ee-e145a3077f13","created":"2021-09-06T12:43:07.681365+0000","modified":"2021-09-06T12:43:25.680178+0000","last_up_change":"2021-09-06T12:43:15.808870+0000","last_in_change":"2021-09-06T12:43:08.652717+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2021-09-06T12:43:14.841634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-09-06T12:43:22.751582+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"73861d13-b3af-4fc8-b5d5-2911da2b6f9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":31600},{"type":"v1","addr":"172.21.15.52:6801","nonce":31600}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":31600},{"type":"v1","addr":"172.21.15.52:6803","nonce":31600}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":31600},{"type":"v1","addr":"172.21.15.52:6807","nonce":31600}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":31600},{"type":"v1","addr":"172.21.15.52:6805","nonce":31600}]},"public_addr":"172.21.15.52:6801/31600","cluster_addr":"172.21.15.52:6803/31600","heartbeat_back_addr":"172.21.15.52:6807/31600","heartbeat_front_addr":"172.21.15.52:6805/31600","state":["exists","up"]},{"osd":1,"uuid":"de1d9fa3-a1f7-43f9-9bff-3b0251a52399","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":31601},{"type":"v1","addr":"172.21.15.52:6809","nonce":31601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":31601},{"type":"v1","addr":"172.21.15.52:6811","nonce":31601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6815","nonce":31601},{"type":"v1","addr":"172.21.15.52:6817","nonce":31601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":31601},{"type":"v1","addr":"172.21.15.52:6813","nonce":31601}]},"public_addr":"172.21.15.52:6809/31601","cluster_addr":"172.21.15.52:6811/31601","heartbeat_back_addr":"172.21.15.52:6817/31601","heartbeat_front_addr":"172.21.15.52:6813/31601","state":["exists","up"]},{"osd":2,"uuid":"8b47c2ac-d408-4381-abaf-413e33f6d6c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":31602},{"type":"v1","addr":"172.21.15.52:6816","nonce":31602}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":31602},{"type":"v1","addr":"172.21.15.52:6819","nonce":31602}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":31602},{"type":"v1","addr":"172.21.15.52:6823","nonce":31602}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":31602},{"type":"v1","addr":"172.21.15.52:6821","nonce":31602}]},"public_addr":"172.21.15.52:6816/31602","cluster_addr":"172.21.15.52:6819/31602","heartbeat_back_addr":"172.21.15.52:6823/31602","heartbeat_front_addr":"172.21.15.52:6821/31602","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790726+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.991216+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-09-06T12:43:12.790156+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-09-06T12:43:26.652 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-09-06T12:43:26.802 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2021-09-06T12:43:26.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-09-06T12:43:26.952 INFO:teuthology.orchestra.run.smithi052.stdout:42949672967 2021-09-06T12:43:26.953 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-09-06T12:43:27.102 INFO:teuthology.orchestra.run.smithi052.stdout:34359738373 2021-09-06T12:43:27.102 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-06T12:43:27.403 INFO:teuthology.orchestra.run.smithi052.stdout:34359738370 2021-09-06T12:43:27.416 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738370 for osd.0 2021-09-06T12:43:28.417 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-09-06T12:43:28.731 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2021-09-06T12:43:28.744 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.0 2021-09-06T12:43:28.745 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-06T12:43:29.048 INFO:teuthology.orchestra.run.smithi052.stdout:42949672965 2021-09-06T12:43:29.061 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672967 got 42949672965 for osd.1 2021-09-06T12:43:30.062 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-09-06T12:43:30.379 INFO:teuthology.orchestra.run.smithi052.stdout:42949672967 2021-09-06T12:43:30.392 INFO:tasks.ceph.ceph_manager.c2:need seq 42949672967 got 42949672967 for osd.1 2021-09-06T12:43:30.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-09-06T12:43:30.693 INFO:teuthology.orchestra.run.smithi052.stdout:34359738373 2021-09-06T12:43:30.706 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738373 got 34359738373 for osd.2 2021-09-06T12:43:30.707 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-09-06T12:43:30.707 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-06T12:43:31.004 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:31.006 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2021-09-06T12:43:31.017 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-09-06T12:43:30.796773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24800,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":314548000,"statfs":{"total":322122547200,"available":322097152000,"internally_reserved":0,"allocated":1581056,"data_stored":1291361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.120462"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.687235+0000","last_change":"2021-09-06T12:43:24.685412+0000","last_active":"2021-09-06T12:43:24.687235+0000","last_peered":"2021-09-06T12:43:24.687235+0000","last_clean":"2021-09-06T12:43:24.687235+0000","last_became_active":"2021-09-06T12:43:24.685092+0000","last_became_peered":"2021-09-06T12:43:24.685092+0000","last_unstale":"2021-09-06T12:43:24.687235+0000","last_undegraded":"2021-09-06T12:43:24.687235+0000","last_fullsized":"2021-09-06T12:43:24.687235+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686861+0000","last_change":"2021-09-06T12:43:24.684183+0000","last_active":"2021-09-06T12:43:24.686861+0000","last_peered":"2021-09-06T12:43:24.686861+0000","last_clean":"2021-09-06T12:43:24.686861+0000","last_became_active":"2021-09-06T12:43:24.683833+0000","last_became_peered":"2021-09-06T12:43:24.683833+0000","last_unstale":"2021-09-06T12:43:24.686861+0000","last_undegraded":"2021-09-06T12:43:24.686861+0000","last_fullsized":"2021-09-06T12:43:24.686861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686933+0000","last_change":"2021-09-06T12:43:24.684044+0000","last_active":"2021-09-06T12:43:24.686933+0000","last_peered":"2021-09-06T12:43:24.686933+0000","last_clean":"2021-09-06T12:43:24.686933+0000","last_became_active":"2021-09-06T12:43:24.683596+0000","last_became_peered":"2021-09-06T12:43:24.683596+0000","last_unstale":"2021-09-06T12:43:24.686933+0000","last_undegraded":"2021-09-06T12:43:24.686933+0000","last_fullsized":"2021-09-06T12:43:24.686933+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.685687+0000","last_change":"2021-09-06T12:43:24.684950+0000","last_active":"2021-09-06T12:43:24.685687+0000","last_peered":"2021-09-06T12:43:24.685687+0000","last_clean":"2021-09-06T12:43:24.685687+0000","last_became_active":"2021-09-06T12:43:24.684628+0000","last_became_peered":"2021-09-06T12:43:24.684628+0000","last_unstale":"2021-09-06T12:43:24.685687+0000","last_undegraded":"2021-09-06T12:43:24.685687+0000","last_fullsized":"2021-09-06T12:43:24.685687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.678193+0000","last_change":"2021-09-06T12:43:17.739608+0000","last_active":"2021-09-06T12:43:24.678193+0000","last_peered":"2021-09-06T12:43:24.678193+0000","last_clean":"2021-09-06T12:43:24.678193+0000","last_became_active":"2021-09-06T12:43:17.738400+0000","last_became_peered":"2021-09-06T12:43:17.738400+0000","last_unstale":"2021-09-06T12:43:24.678193+0000","last_undegraded":"2021-09-06T12:43:24.678193+0000","last_fullsized":"2021-09-06T12:43:24.678193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:15.808870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:15.808870+0000","last_clean_scrub_stamp":"2021-09-06T12:43:15.808870+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686894+0000","last_change":"2021-09-06T12:43:24.684800+0000","last_active":"2021-09-06T12:43:24.686894+0000","last_peered":"2021-09-06T12:43:24.686894+0000","last_clean":"2021-09-06T12:43:24.686894+0000","last_became_active":"2021-09-06T12:43:24.684474+0000","last_became_peered":"2021-09-06T12:43:24.684474+0000","last_unstale":"2021-09-06T12:43:24.686894+0000","last_undegraded":"2021-09-06T12:43:24.686894+0000","last_fullsized":"2021-09-06T12:43:24.686894+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.687096+0000","last_change":"2021-09-06T12:43:24.685833+0000","last_active":"2021-09-06T12:43:24.687096+0000","last_peered":"2021-09-06T12:43:24.687096+0000","last_clean":"2021-09-06T12:43:24.687096+0000","last_became_active":"2021-09-06T12:43:24.685363+0000","last_became_peered":"2021-09-06T12:43:24.685363+0000","last_unstale":"2021-09-06T12:43:24.687096+0000","last_undegraded":"2021-09-06T12:43:24.687096+0000","last_fullsized":"2021-09-06T12:43:24.687096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.688195+0000","last_change":"2021-09-06T12:43:24.687627+0000","last_active":"2021-09-06T12:43:24.688195+0000","last_peered":"2021-09-06T12:43:24.688195+0000","last_clean":"2021-09-06T12:43:24.688195+0000","last_became_active":"2021-09-06T12:43:24.687307+0000","last_became_peered":"2021-09-06T12:43:24.687307+0000","last_unstale":"2021-09-06T12:43:24.688195+0000","last_undegraded":"2021-09-06T12:43:24.688195+0000","last_fullsized":"2021-09-06T12:43:24.688195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686638+0000","last_change":"2021-09-06T12:43:24.684875+0000","last_active":"2021-09-06T12:43:24.686638+0000","last_peered":"2021-09-06T12:43:24.686638+0000","last_clean":"2021-09-06T12:43:24.686638+0000","last_became_active":"2021-09-06T12:43:24.684537+0000","last_became_peered":"2021-09-06T12:43:24.684537+0000","last_unstale":"2021-09-06T12:43:24.686638+0000","last_undegraded":"2021-09-06T12:43:24.686638+0000","last_fullsized":"2021-09-06T12:43:24.686638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8588,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849012,"statfs":{"total":107374182400,"available":107365388288,"internally_reserved":0,"allocated":724992,"data_stored":627243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8588,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849012,"statfs":{"total":107374182400,"available":107365388288,"internally_reserved":0,"allocated":724992,"data_stored":627243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7624,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104849976,"statfs":{"total":107374182400,"available":107366375424,"internally_reserved":0,"allocated":131072,"data_stored":36875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T12:43:31.018 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-09-06T12:43:31.311 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:31.313 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2021-09-06T12:43:31.324 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2021-09-06T12:43:30.796773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":24800,"kb_used_data":1544,"kb_used_omap":0,"kb_used_meta":23232,"kb_avail":314548000,"statfs":{"total":322122547200,"available":322097152000,"internally_reserved":0,"allocated":1581056,"data_stored":1291361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":23789568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.120462"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.687235+0000","last_change":"2021-09-06T12:43:24.685412+0000","last_active":"2021-09-06T12:43:24.687235+0000","last_peered":"2021-09-06T12:43:24.687235+0000","last_clean":"2021-09-06T12:43:24.687235+0000","last_became_active":"2021-09-06T12:43:24.685092+0000","last_became_peered":"2021-09-06T12:43:24.685092+0000","last_unstale":"2021-09-06T12:43:24.687235+0000","last_undegraded":"2021-09-06T12:43:24.687235+0000","last_fullsized":"2021-09-06T12:43:24.687235+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686861+0000","last_change":"2021-09-06T12:43:24.684183+0000","last_active":"2021-09-06T12:43:24.686861+0000","last_peered":"2021-09-06T12:43:24.686861+0000","last_clean":"2021-09-06T12:43:24.686861+0000","last_became_active":"2021-09-06T12:43:24.683833+0000","last_became_peered":"2021-09-06T12:43:24.683833+0000","last_unstale":"2021-09-06T12:43:24.686861+0000","last_undegraded":"2021-09-06T12:43:24.686861+0000","last_fullsized":"2021-09-06T12:43:24.686861+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686933+0000","last_change":"2021-09-06T12:43:24.684044+0000","last_active":"2021-09-06T12:43:24.686933+0000","last_peered":"2021-09-06T12:43:24.686933+0000","last_clean":"2021-09-06T12:43:24.686933+0000","last_became_active":"2021-09-06T12:43:24.683596+0000","last_became_peered":"2021-09-06T12:43:24.683596+0000","last_unstale":"2021-09-06T12:43:24.686933+0000","last_undegraded":"2021-09-06T12:43:24.686933+0000","last_fullsized":"2021-09-06T12:43:24.686933+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.685687+0000","last_change":"2021-09-06T12:43:24.684950+0000","last_active":"2021-09-06T12:43:24.685687+0000","last_peered":"2021-09-06T12:43:24.685687+0000","last_clean":"2021-09-06T12:43:24.685687+0000","last_became_active":"2021-09-06T12:43:24.684628+0000","last_became_peered":"2021-09-06T12:43:24.684628+0000","last_unstale":"2021-09-06T12:43:24.685687+0000","last_undegraded":"2021-09-06T12:43:24.685687+0000","last_fullsized":"2021-09-06T12:43:24.685687+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.678193+0000","last_change":"2021-09-06T12:43:17.739608+0000","last_active":"2021-09-06T12:43:24.678193+0000","last_peered":"2021-09-06T12:43:24.678193+0000","last_clean":"2021-09-06T12:43:24.678193+0000","last_became_active":"2021-09-06T12:43:17.738400+0000","last_became_peered":"2021-09-06T12:43:17.738400+0000","last_unstale":"2021-09-06T12:43:24.678193+0000","last_undegraded":"2021-09-06T12:43:24.678193+0000","last_fullsized":"2021-09-06T12:43:24.678193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:15.808870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:15.808870+0000","last_clean_scrub_stamp":"2021-09-06T12:43:15.808870+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686894+0000","last_change":"2021-09-06T12:43:24.684800+0000","last_active":"2021-09-06T12:43:24.686894+0000","last_peered":"2021-09-06T12:43:24.686894+0000","last_clean":"2021-09-06T12:43:24.686894+0000","last_became_active":"2021-09-06T12:43:24.684474+0000","last_became_peered":"2021-09-06T12:43:24.684474+0000","last_unstale":"2021-09-06T12:43:24.686894+0000","last_undegraded":"2021-09-06T12:43:24.686894+0000","last_fullsized":"2021-09-06T12:43:24.686894+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.687096+0000","last_change":"2021-09-06T12:43:24.685833+0000","last_active":"2021-09-06T12:43:24.687096+0000","last_peered":"2021-09-06T12:43:24.687096+0000","last_clean":"2021-09-06T12:43:24.687096+0000","last_became_active":"2021-09-06T12:43:24.685363+0000","last_became_peered":"2021-09-06T12:43:24.685363+0000","last_unstale":"2021-09-06T12:43:24.687096+0000","last_undegraded":"2021-09-06T12:43:24.687096+0000","last_fullsized":"2021-09-06T12:43:24.687096+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.688195+0000","last_change":"2021-09-06T12:43:24.687627+0000","last_active":"2021-09-06T12:43:24.688195+0000","last_peered":"2021-09-06T12:43:24.688195+0000","last_clean":"2021-09-06T12:43:24.688195+0000","last_became_active":"2021-09-06T12:43:24.687307+0000","last_became_peered":"2021-09-06T12:43:24.687307+0000","last_unstale":"2021-09-06T12:43:24.688195+0000","last_undegraded":"2021-09-06T12:43:24.688195+0000","last_fullsized":"2021-09-06T12:43:24.688195+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2021-09-06T12:43:24.686638+0000","last_change":"2021-09-06T12:43:24.684875+0000","last_active":"2021-09-06T12:43:24.686638+0000","last_peered":"2021-09-06T12:43:24.686638+0000","last_clean":"2021-09-06T12:43:24.686638+0000","last_became_active":"2021-09-06T12:43:24.684537+0000","last_became_peered":"2021-09-06T12:43:24.684537+0000","last_unstale":"2021-09-06T12:43:24.686638+0000","last_undegraded":"2021-09-06T12:43:24.686638+0000","last_fullsized":"2021-09-06T12:43:24.686638+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-09-06T12:43:23.673545+0000","last_clean_scrub_stamp":"2021-09-06T12:43:23.673545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8588,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849012,"statfs":{"total":107374182400,"available":107365388288,"internally_reserved":0,"allocated":724992,"data_stored":627243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":8588,"kb_used_data":708,"kb_used_omap":0,"kb_used_meta":7872,"kb_avail":104849012,"statfs":{"total":107374182400,"available":107365388288,"internally_reserved":0,"allocated":724992,"data_stored":627243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8060928},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]}]},{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":7624,"kb_used_data":128,"kb_used_omap":0,"kb_used_meta":7488,"kb_avail":104849976,"statfs":{"total":107374182400,"available":107366375424,"internally_reserved":0,"allocated":131072,"data_stored":36875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":7667712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-09-06T12:43:31.325 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-09-06T12:43:31.325 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-09-06T12:43:31.325 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-09-06T12:43:31.326 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-09-06T12:43:31.695 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-09-06T12:43:31.695 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-09-06T12:43:31.707 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-09-06T12:43:31.708 INFO:teuthology.run_tasks:Running task rgw... 2021-09-06T12:43:31.722 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-09-06T12:43:31.723 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-09-06T12:43:31.723 INFO:tasks.rgw:Creating data pools 2021-09-06T12:43:31.724 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-09-06T12:43:31.724 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-06T12:43:32.766 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'default.rgw.buckets.data' created 2021-09-06T12:43:32.784 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-06T12:43:33.778 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T12:43:33.797 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-06T12:43:34.778 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'default.rgw.buckets.index' created 2021-09-06T12:43:34.796 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-06T12:43:35.791 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T12:43:35.810 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-09-06T12:43:35.810 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-09-06T12:43:36.110 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-06T12:43:36.121 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-09-06T12:43:36.778 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T12:43:36.789 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-09-06T12:43:37.084 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-06T12:43:37.095 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-09-06T12:43:37.782 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T12:43:37.793 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-09-06T12:43:37.794 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-06T12:43:38.863 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.data' created 2021-09-06T12:43:38.881 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-06T12:43:39.882 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T12:43:39.903 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-06T12:43:40.866 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.index' created 2021-09-06T12:43:40.888 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-06T12:43:41.902 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T12:43:41.915 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-09-06T12:43:41.916 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-09-06T12:43:42.215 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.data' already exists 2021-09-06T12:43:42.225 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-09-06T12:43:42.869 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-09-06T12:43:42.880 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-09-06T12:43:43.174 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.index' already exists 2021-09-06T12:43:43.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-09-06T12:43:43.871 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-09-06T12:43:43.883 DEBUG:tasks.rgw:Pools created 2021-09-06T12:43:43.884 INFO:tasks.rgw:Configuring compression type = random 2021-09-06T12:43:43.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-09-06T12:43:43.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-09-06T12:43:43.885 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-09-06T12:43:43.953 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:43:43.954 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:43:43.960 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:43.961+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.817 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.818+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.817 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.818+0000 7fda6df21380 20 realm 2021-09-06T12:43:45.818 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.818+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.818 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.819+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.818 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.819+0000 7fda6df21380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T12:43:45.819 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.819+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.819 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.820+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.819 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.820+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.820 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.821+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.820 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.821+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.821 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.822+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.821 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.822+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.822 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.823+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.822 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.823+0000 7fda6df21380 10 Creating default zonegroup 2021-09-06T12:43:45.822 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.823+0000 7fda6df21380 20 get_system_obj_state: rctx=0x7ffd303de430 obj=:.pools.avail state=0x558a2fb45d70 s->prefetch_data=0 2021-09-06T12:43:45.823 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.824+0000 7fda6df21380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-06T12:43:45.839 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.840+0000 7fda6df21380 10 WARNING: store->list_zones() returned r=-2 2021-09-06T12:43:45.839 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.840+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.839 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.840+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.843 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.844+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.844 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.845+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.844 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.845+0000 7fda6df21380 10 could not read realm id: (2) No such file or directory 2021-09-06T12:43:45.844 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.845+0000 7fda6df21380 10 WARNING: failed to set zone as default, r=-22 2021-09-06T12:43:45.845 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.845+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.845 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.845+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.847 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.848+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.847 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.849+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:45.848 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.849+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.850+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:45.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.850+0000 7fda6df21380 10 Using default name default 2021-09-06T12:43:45.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.850+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.851+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:45.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.851+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.851+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:45.851 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.851+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.852 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.852+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.852 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.852+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.852 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.852+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.853 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.852+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.853 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.852+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:45.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.863+0000 7fda6df21380 20 RGWRados::pool_iterate: got zone_info.b8b4ef92-b205-4874-ac8c-4da1545b13f7 2021-09-06T12:43:45.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.863+0000 7fda6df21380 20 RGWRados::pool_iterate: got zonegroup_info.e90d8aed-b3f5-4548-80af-dbd07a8a6724 2021-09-06T12:43:45.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.863+0000 7fda6df21380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:45.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.863+0000 7fda6df21380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:45.864 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.863+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.864 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.864+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:45.864 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.864+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.864 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.864+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:45.865 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.864+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.866 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:45.866 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:45.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 zone default found 2021-09-06T12:43:45.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 1 Realm: () 2021-09-06T12:43:45.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 1 ZoneGroup: default (e90d8aed-b3f5-4548-80af-dbd07a8a6724) 2021-09-06T12:43:45.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 1 Zone: default (b8b4ef92-b205-4874-ac8c-4da1545b13f7) 2021-09-06T12:43:45.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T12:43:45.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 zonegroup default 2021-09-06T12:43:45.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.865+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:45.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.866+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:45.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:45.866+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:47.815 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:47.816+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:47.815 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:47.816+0000 7fda6df21380 20 rados->read ofs=0 len=0 2021-09-06T12:43:47.816 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:47.817+0000 7fda6df21380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:47.817 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:47.817+0000 7fda6df21380 20 started sync module instance, tier type = 2021-09-06T12:43:47.817 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:47.817+0000 7fda6df21380 20 started zone id=b8b4ef92-b205-4874-ac8c-4da1545b13f7 (name=default) with tier type = 2021-09-06T12:43:49.870 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=0 2021-09-06T12:43:49.870 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=1 2021-09-06T12:43:49.871 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=2 2021-09-06T12:43:49.871 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=3 2021-09-06T12:43:49.871 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=4 2021-09-06T12:43:49.871 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=5 2021-09-06T12:43:49.872 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.871+0000 7fda6df21380 20 add_watcher() i=6 2021-09-06T12:43:49.872 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 add_watcher() i=7 2021-09-06T12:43:49.872 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 2 all 8 watchers are set, enabling cache 2021-09-06T12:43:49.873 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-06T12:43:49.873 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-06T12:43:49.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:49.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-06T12:43:49.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-06T12:43:49.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.874+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:49.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-06T12:43:49.875 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-06T12:43:49.875 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:49.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-06T12:43:49.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-06T12:43:49.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:49.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-06T12:43:49.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.875+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-06T12:43:49.877 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.877+0000 7fda6df21380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:49.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.883+0000 7fda327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:43:49.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.883+0000 7fda6df21380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:43:49.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:49.883+0000 7fda6df21380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:43:53.832 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.831+0000 7fda6df21380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-06T12:43:53.833 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.831+0000 7fda6df21380 5 note: GC not initialized 2021-09-06T12:43:53.834 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.833+0000 7fd9a17fa700 20 reqs_thread_entry: start 2021-09-06T12:43:53.834 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.834+0000 7fd98effd700 20 reqs_thread_entry: start 2021-09-06T12:43:53.834 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.834+0000 7fda6df21380 20 init_complete bucket index max shards: 11 2021-09-06T12:43:53.835 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.834+0000 7fda6df21380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T12:43:53.835 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.834+0000 7fd98d7fa700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 12:44:24 2021 2021-09-06T12:43:53.835 INFO:teuthology.orchestra.run.smithi041.stderr: 2021-09-06T12:43:53.835 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.834+0000 7fd98cff9700 20 reqs_thread_entry: start 2021-09-06T12:43:53.862 INFO:teuthology.orchestra.run.smithi041.stdout:[] 2021-09-06T12:43:53.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.863+0000 7fda6df21380 20 remove_watcher() i=0 2021-09-06T12:43:53.863 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.863+0000 7fda6df21380 2 removed watcher, disabling cache 2021-09-06T12:43:53.864 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.864+0000 7fda6df21380 20 remove_watcher() i=1 2021-09-06T12:43:53.866 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.867+0000 7fda6df21380 20 remove_watcher() i=2 2021-09-06T12:43:53.869 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.869+0000 7fda6df21380 20 remove_watcher() i=3 2021-09-06T12:43:53.871 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.872+0000 7fda6df21380 20 remove_watcher() i=4 2021-09-06T12:43:53.873 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.873+0000 7fda6df21380 20 remove_watcher() i=5 2021-09-06T12:43:53.874 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.875+0000 7fda6df21380 20 remove_watcher() i=6 2021-09-06T12:43:53.875 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.876+0000 7fda6df21380 20 remove_watcher() i=7 2021-09-06T12:43:53.876 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.877+0000 7fda327e4700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:43:53.882 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T12:43:53.882 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:43:53.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:43:53.883 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T12:43:53.929 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:43:53.930 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:43:53.935 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.936+0000 7f31c2da4380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:43:53.936 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.936+0000 7f31c2da4380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:43:53.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.940+0000 7f30ff7fe700 20 reqs_thread_entry: start 2021-09-06T12:43:53.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.940+0000 7f31c2da4380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T12:43:53.940 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.940+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.940 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.941+0000 7f31c2da4380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:53.941 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.941+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.941 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.941+0000 7f31c2da4380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:53.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.953+0000 7f31c2da4380 20 RGWRados::pool_iterate: got zone_info.b8b4ef92-b205-4874-ac8c-4da1545b13f7 2021-09-06T12:43:53.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.953+0000 7f31c2da4380 20 RGWRados::pool_iterate: got zonegroup_info.e90d8aed-b3f5-4548-80af-dbd07a8a6724 2021-09-06T12:43:53.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.953+0000 7f31c2da4380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:53.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.953+0000 7f31c2da4380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:53.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.953+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:53.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:53.956 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.956 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:53.957 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.954+0000 7f31c2da4380 20 rados->read ofs=0 len=0 2021-09-06T12:43:53.957 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:53.955+0000 7f31c2da4380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:53.959 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:43:53.959 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "b8b4ef92-b205-4874-ac8c-4da1545b13f7", 2021-09-06T12:43:53.959 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default", 2021-09-06T12:43:53.959 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T12:43:53.960 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "default.rgw.control", 2021-09-06T12:43:53.960 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T12:43:53.960 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T12:43:53.960 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "default.rgw.log", 2021-09-06T12:43:53.960 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T12:43:53.961 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T12:43:53.961 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T12:43:53.961 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T12:43:53.961 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T12:43:53.962 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T12:43:53.962 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T12:43:53.962 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T12:43:53.962 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T12:43:53.962 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:43:53.963 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "", 2021-09-06T12:43:53.963 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "" 2021-09-06T12:43:53.963 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:43:53.963 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:43:53.964 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:43:53.964 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:43:53.964 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:43:53.964 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T12:43:53.964 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:43:53.965 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:43:53.965 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T12:43:53.965 INFO:teuthology.orchestra.run.smithi041.stdout: "compression_type": "random" 2021-09-06T12:43:53.965 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:53.965 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:43:53.966 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T12:43:53.966 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:43:53.966 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:53.966 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:53.966 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:43:53.967 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "", 2021-09-06T12:43:53.967 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T12:43:53.967 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:43:53.968 DEBUG:tasks.util.rgw: json result: {'id': 'b8b4ef92-b205-4874-ac8c-4da1545b13f7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T12:43:53.968 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-09-06T12:43:53.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-09-06T12:43:53.969 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-09-06T12:43:54.007 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:43:54.007 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:43:54.013 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.013+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 realm 2021-09-06T12:43:54.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.016 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T12:43:54.016 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.016 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.016 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.015+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.017 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.016+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.017 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.016+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.018 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.016+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:43:54.018 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.016+0000 7fc6cfbd0380 20 searching for the correct realm 2021-09-06T12:43:54.027 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zone_info.b8b4ef92-b205-4874-ac8c-4da1545b13f7 2021-09-06T12:43:54.027 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zonegroup_info.e90d8aed-b3f5-4548-80af-dbd07a8a6724 2021-09-06T12:43:54.027 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:54.028 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:54.028 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.028 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.028 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.028 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.029 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.027+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.030 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:54.030 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.030 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.030 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.031 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.031 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.028+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.031 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.029+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.038+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zone_info.b8b4ef92-b205-4874-ac8c-4da1545b13f7 2021-09-06T12:43:54.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.038+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zonegroup_info.e90d8aed-b3f5-4548-80af-dbd07a8a6724 2021-09-06T12:43:54.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.038+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:54.039 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.038+0000 7fc6cfbd0380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:54.039 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.038+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.041 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.041 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.041 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:43:54.042 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.042 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.042 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.039+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.042 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:54.043 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.043 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.043 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 zone default found 2021-09-06T12:43:54.043 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 1 Realm: () 2021-09-06T12:43:54.043 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 1 ZoneGroup: default (e90d8aed-b3f5-4548-80af-dbd07a8a6724) 2021-09-06T12:43:54.044 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 1 Zone: default (b8b4ef92-b205-4874-ac8c-4da1545b13f7) 2021-09-06T12:43:54.044 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T12:43:54.044 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 zonegroup default 2021-09-06T12:43:54.044 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.045 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.045 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.040+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.045 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.041+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.045 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.041+0000 7fc6cfbd0380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.045 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.041+0000 7fc6cfbd0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.046 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.041+0000 7fc6cfbd0380 20 started sync module instance, tier type = 2021-09-06T12:43:54.046 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.041+0000 7fc6cfbd0380 20 started zone id=b8b4ef92-b205-4874-ac8c-4da1545b13f7 (name=default) with tier type = 2021-09-06T12:43:54.061 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=0 2021-09-06T12:43:54.061 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=1 2021-09-06T12:43:54.061 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=2 2021-09-06T12:43:54.061 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=3 2021-09-06T12:43:54.062 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=4 2021-09-06T12:43:54.062 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=5 2021-09-06T12:43:54.062 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.061+0000 7fc6cfbd0380 20 add_watcher() i=6 2021-09-06T12:43:54.063 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.063+0000 7fc6cfbd0380 20 add_watcher() i=7 2021-09-06T12:43:54.063 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.063+0000 7fc6cfbd0380 2 all 8 watchers are set, enabling cache 2021-09-06T12:43:54.067 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.067+0000 7fc60ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:43:54.067 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.067+0000 7fc6cfbd0380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:43:54.067 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.067+0000 7fc6cfbd0380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:43:54.070 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.071+0000 7fc6cfbd0380 5 note: GC not initialized 2021-09-06T12:43:54.072 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc606ffd700 20 reqs_thread_entry: start 2021-09-06T12:43:54.072 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc604ff9700 20 reqs_thread_entry: start 2021-09-06T12:43:54.072 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc6cfbd0380 20 init_complete bucket index max shards: 11 2021-09-06T12:43:54.073 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc6cfbd0380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T12:43:54.073 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc5f2ffd700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 12:44:24 2021 2021-09-06T12:43:54.073 INFO:teuthology.orchestra.run.smithi041.stderr: 2021-09-06T12:43:54.073 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.072+0000 7fc5f27fc700 20 reqs_thread_entry: start 2021-09-06T12:43:54.083 INFO:teuthology.orchestra.run.smithi041.stdout:[] 2021-09-06T12:43:54.083 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.083+0000 7fc6cfbd0380 20 remove_watcher() i=0 2021-09-06T12:43:54.084 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.083+0000 7fc6cfbd0380 2 removed watcher, disabling cache 2021-09-06T12:43:54.084 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.085+0000 7fc6cfbd0380 20 remove_watcher() i=1 2021-09-06T12:43:54.085 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.086+0000 7fc6cfbd0380 20 remove_watcher() i=2 2021-09-06T12:43:54.088 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.088+0000 7fc6cfbd0380 20 remove_watcher() i=3 2021-09-06T12:43:54.089 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.089+0000 7fc6cfbd0380 20 remove_watcher() i=4 2021-09-06T12:43:54.090 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.091+0000 7fc6cfbd0380 20 remove_watcher() i=5 2021-09-06T12:43:54.093 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.094+0000 7fc6cfbd0380 20 remove_watcher() i=6 2021-09-06T12:43:54.094 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.095+0000 7fc6cfbd0380 20 remove_watcher() i=7 2021-09-06T12:43:54.095 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.095+0000 7fc60ffff700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:43:54.100 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T12:43:54.100 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:43:54.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:43:54.101 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T12:43:54.184 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:43:54.185 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:43:54.190 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.191+0000 7f9b38e7e380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:43:54.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.191+0000 7f9b38e7e380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:43:54.194 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.194+0000 7f9ab57fa700 20 reqs_thread_entry: start 2021-09-06T12:43:54.194 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.194+0000 7f9b38e7e380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T12:43:54.194 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.194+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.196 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.195+0000 7f9b38e7e380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.196 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.195+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.196 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.196+0000 7f9b38e7e380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:43:54.207 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 RGWRados::pool_iterate: got zone_info.b8b4ef92-b205-4874-ac8c-4da1545b13f7 2021-09-06T12:43:54.207 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 RGWRados::pool_iterate: got zonegroup_info.e90d8aed-b3f5-4548-80af-dbd07a8a6724 2021-09-06T12:43:54.208 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:54.208 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:54.208 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.208 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:43:54.209 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.207+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.209 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.208+0000 7f9b38e7e380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:54.209 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.208+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.210 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.208+0000 7f9b38e7e380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:54.210 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.208+0000 7f9b38e7e380 20 rados->read ofs=0 len=0 2021-09-06T12:43:54.210 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:43:54.208+0000 7f9b38e7e380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:54.212 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:43:54.212 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "b8b4ef92-b205-4874-ac8c-4da1545b13f7", 2021-09-06T12:43:54.213 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default", 2021-09-06T12:43:54.213 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T12:43:54.213 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "default.rgw.control", 2021-09-06T12:43:54.213 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T12:43:54.214 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T12:43:54.214 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "default.rgw.log", 2021-09-06T12:43:54.214 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T12:43:54.214 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T12:43:54.214 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T12:43:54.215 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T12:43:54.215 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T12:43:54.215 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T12:43:54.215 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T12:43:54.216 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T12:43:54.216 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T12:43:54.216 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:43:54.216 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "", 2021-09-06T12:43:54.216 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "" 2021-09-06T12:43:54.217 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:43:54.217 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:43:54.217 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:43:54.217 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:43:54.218 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:43:54.218 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T12:43:54.218 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:43:54.218 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:43:54.218 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T12:43:54.219 INFO:teuthology.orchestra.run.smithi041.stdout: "compression_type": "random" 2021-09-06T12:43:54.219 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:54.219 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:43:54.219 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T12:43:54.219 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:43:54.220 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:54.220 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:43:54.220 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:43:54.220 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "", 2021-09-06T12:43:54.220 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T12:43:54.221 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:43:54.222 DEBUG:tasks.util.rgw: json result: {'id': 'b8b4ef92-b205-4874-ac8c-4da1545b13f7', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T12:43:54.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-09-06T12:43:54.223 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-09-06T12:43:54.223 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-09-06T12:43:54.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:43:54.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:43:54.314 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:54.315+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.698 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.699+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.698 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.699+0000 7fa707ffd380 20 realm 2021-09-06T12:43:55.698 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.699+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.699 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.700+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.699 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.700+0000 7fa707ffd380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T12:43:55.699 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.700+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.702 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.704+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.703 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.704+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.703 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.705+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.704 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.705+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.704 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.705+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.704 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.705+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.705 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.706+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.705 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.706+0000 7fa707ffd380 10 Creating default zonegroup 2021-09-06T12:43:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.706+0000 7fa707ffd380 20 get_system_obj_state: rctx=0x7ffd54397a80 obj=:.pools.avail state=0x562981a64d70 s->prefetch_data=0 2021-09-06T12:43:55.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.707+0000 7fa707ffd380 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-09-06T12:43:55.723 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.724+0000 7fa707ffd380 10 WARNING: store->list_zones() returned r=-2 2021-09-06T12:43:55.723 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.724+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.723 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.725+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 10 could not read realm id: (2) No such file or directory 2021-09-06T12:43:55.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 10 WARNING: failed to set zone as default, r=-22 2021-09-06T12:43:55.727 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.728 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.728+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.731+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.732+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:55.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.732+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.732 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.732+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:55.732 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.732+0000 7fa707ffd380 10 Using default name default 2021-09-06T12:43:55.732 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.732+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:55.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:55.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.734 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.733+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.734 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.734+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.734 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.734+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.735 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.734+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:55.746 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 RGWRados::pool_iterate: got zonegroup_info.6179ff01-2837-43c7-8636-bb86386c205f 2021-09-06T12:43:55.746 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 RGWRados::pool_iterate: got zone_info.5db7c9f1-7199-4792-8f33-b1c03046b437 2021-09-06T12:43:55.746 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:43:55.747 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:43:55.747 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.747 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:55.747 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.747 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:43:55.748 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.747+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.749 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.748+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:43:55.749 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.748+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.749 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.748+0000 7fa707ffd380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:43:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.748+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 zone default found 2021-09-06T12:43:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 1 Realm: () 2021-09-06T12:43:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 1 ZoneGroup: default (6179ff01-2837-43c7-8636-bb86386c205f) 2021-09-06T12:43:55.751 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 1 Zone: default (5db7c9f1-7199-4792-8f33-b1c03046b437) 2021-09-06T12:43:55.751 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T12:43:55.751 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 zonegroup default 2021-09-06T12:43:55.751 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:55.751 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:55.752 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:55.749+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:57.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:57.707+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:57.707 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:57.707+0000 7fa707ffd380 20 rados->read ofs=0 len=0 2021-09-06T12:43:57.707 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:57.707+0000 7fa707ffd380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:43:57.708 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:57.707+0000 7fa707ffd380 20 started sync module instance, tier type = 2021-09-06T12:43:57.708 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:57.707+0000 7fa707ffd380 20 started zone id=5db7c9f1-7199-4792-8f33-b1c03046b437 (name=default) with tier type = 2021-09-06T12:43:59.735 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=0 2021-09-06T12:43:59.735 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=1 2021-09-06T12:43:59.735 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=2 2021-09-06T12:43:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=3 2021-09-06T12:43:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=4 2021-09-06T12:43:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=5 2021-09-06T12:43:59.736 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.735+0000 7fa707ffd380 20 add_watcher() i=6 2021-09-06T12:43:59.738 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.736+0000 7fa707ffd380 20 add_watcher() i=7 2021-09-06T12:43:59.738 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.736+0000 7fa707ffd380 2 all 8 watchers are set, enabling cache 2021-09-06T12:43:59.738 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2021-09-06T12:43:59.739 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2021-09-06T12:43:59.739 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:59.739 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2021-09-06T12:43:59.739 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2021-09-06T12:43:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2021-09-06T12:43:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2021-09-06T12:43:59.740 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.737+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:59.741 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.738+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2021-09-06T12:43:59.741 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.738+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2021-09-06T12:43:59.741 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.738+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:59.741 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.738+0000 7fa707ffd380 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2021-09-06T12:43:59.741 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.738+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1203 entering 2021-09-06T12:43:59.742 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.739+0000 7fa707ffd380 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr*, optional_yield, std::optional, bool):1161 entering 2021-09-06T12:43:59.743 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.743+0000 7fa6ccfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:43:59.743 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.743+0000 7fa707ffd380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:43:59.743 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:43:59.743+0000 7fa707ffd380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:44:03.705 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.704+0000 7fa707ffd380 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-09-06T12:44:03.705 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.704+0000 7fa707ffd380 5 note: GC not initialized 2021-09-06T12:44:03.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.705+0000 7fa62f7fe700 20 reqs_thread_entry: start 2021-09-06T12:44:03.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.705+0000 7fa62d7fa700 20 reqs_thread_entry: start 2021-09-06T12:44:03.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.705+0000 7fa707ffd380 20 init_complete bucket index max shards: 11 2021-09-06T12:44:03.707 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.706+0000 7fa707ffd380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T12:44:03.707 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.706+0000 7fa61b7fe700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 12:44:34 2021 2021-09-06T12:44:03.707 INFO:teuthology.orchestra.run.smithi052.stderr: 2021-09-06T12:44:03.708 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.706+0000 7fa61affd700 20 reqs_thread_entry: start 2021-09-06T12:44:03.729 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-09-06T12:44:03.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.730+0000 7fa707ffd380 20 remove_watcher() i=0 2021-09-06T12:44:03.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.730+0000 7fa707ffd380 2 removed watcher, disabling cache 2021-09-06T12:44:03.730 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.731+0000 7fa707ffd380 20 remove_watcher() i=1 2021-09-06T12:44:03.732 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.733+0000 7fa707ffd380 20 remove_watcher() i=2 2021-09-06T12:44:03.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.734+0000 7fa707ffd380 20 remove_watcher() i=3 2021-09-06T12:44:03.735 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.735+0000 7fa707ffd380 20 remove_watcher() i=4 2021-09-06T12:44:03.736 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.737+0000 7fa707ffd380 20 remove_watcher() i=5 2021-09-06T12:44:03.737 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.738+0000 7fa707ffd380 20 remove_watcher() i=6 2021-09-06T12:44:03.739 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.739+0000 7fa707ffd380 20 remove_watcher() i=7 2021-09-06T12:44:03.740 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.740+0000 7fa6ccfe1700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:44:03.746 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T12:44:03.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:44:03.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:44:03.748 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T12:44:03.796 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:03.796 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:03.802 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.803+0000 7f215b7d0380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:44:03.802 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.803+0000 7f215b7d0380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:44:03.806 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.806+0000 7f2093fff700 20 reqs_thread_entry: start 2021-09-06T12:44:03.806 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.806+0000 7f215b7d0380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T12:44:03.806 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.806+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.807 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.807+0000 7f215b7d0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.807 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.807+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.807 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.808+0000 7f215b7d0380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:44:03.820 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.820+0000 7f215b7d0380 20 RGWRados::pool_iterate: got zonegroup_info.6179ff01-2837-43c7-8636-bb86386c205f 2021-09-06T12:44:03.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.820+0000 7f215b7d0380 20 RGWRados::pool_iterate: got zone_info.5db7c9f1-7199-4792-8f33-b1c03046b437 2021-09-06T12:44:03.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.820+0000 7f215b7d0380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:44:03.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.820+0000 7f215b7d0380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:44:03.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.820+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados_obj.operate() r=0 bl.length=873 2021-09-06T12:44:03.822 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.822 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.822 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.823 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.823 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.821+0000 7f215b7d0380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.823 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.822+0000 7f215b7d0380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:44:03.825 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:03.826 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "5db7c9f1-7199-4792-8f33-b1c03046b437", 2021-09-06T12:44:03.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default", 2021-09-06T12:44:03.826 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T12:44:03.826 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "default.rgw.control", 2021-09-06T12:44:03.827 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T12:44:03.827 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T12:44:03.827 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "default.rgw.log", 2021-09-06T12:44:03.827 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T12:44:03.827 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T12:44:03.828 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T12:44:03.828 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T12:44:03.828 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T12:44:03.828 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T12:44:03.829 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T12:44:03.829 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T12:44:03.829 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T12:44:03.829 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:44:03.829 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "", 2021-09-06T12:44:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "" 2021-09-06T12:44:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:44:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:44:03.831 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:44:03.831 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T12:44:03.831 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:44:03.831 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:44:03.832 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T12:44:03.832 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2021-09-06T12:44:03.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:03.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:03.832 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T12:44:03.833 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:44:03.833 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:03.833 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:03.833 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:03.833 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "", 2021-09-06T12:44:03.834 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T12:44:03.834 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:03.835 DEBUG:tasks.util.rgw: json result: {'id': '5db7c9f1-7199-4792-8f33-b1c03046b437', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T12:44:03.835 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-09-06T12:44:03.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-09-06T12:44:03.836 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-09-06T12:44:03.876 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:03.876 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:03.884 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.885+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.886 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.886 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 20 realm 2021-09-06T12:44:03.886 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.886 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.887 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 0 RGWPeriod::init failed to init realm id : (2) No such file or directory 2021-09-06T12:44:03.887 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.886+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.888 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.887+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.888 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.887+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.889 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.887+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.889 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.887+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.889 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.888+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:44:03.889 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.888+0000 7f31fbab6380 20 searching for the correct realm 2021-09-06T12:44:03.904 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.902+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zonegroup_info.6179ff01-2837-43c7-8636-bb86386c205f 2021-09-06T12:44:03.904 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.902+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zone_info.5db7c9f1-7199-4792-8f33-b1c03046b437 2021-09-06T12:44:03.905 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.902+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:44:03.905 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.902+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:44:03.905 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.902+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.906 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.906 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.907 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.907 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.907 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:44:03.908 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.903+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.908 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.904+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.904+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.904+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.904+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.909 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.904+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.913 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.913+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zonegroup_info.6179ff01-2837-43c7-8636-bb86386c205f 2021-09-06T12:44:03.913 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.913+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zone_info.5db7c9f1-7199-4792-8f33-b1c03046b437 2021-09-06T12:44:03.914 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.913+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:44:03.914 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.913+0000 7f31fbab6380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:44:03.914 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.913+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.914 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.915 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.915 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:44:03.915 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.917 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:03.917 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.914+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.917 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:44:03.918 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.918 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.918 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 zone default found 2021-09-06T12:44:03.918 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 1 Realm: () 2021-09-06T12:44:03.919 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 1 ZoneGroup: default (6179ff01-2837-43c7-8636-bb86386c205f) 2021-09-06T12:44:03.919 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 1 Zone: default (5db7c9f1-7199-4792-8f33-b1c03046b437) 2021-09-06T12:44:03.919 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 10 cannot find current period zonegroup using local zonegroup configuration 2021-09-06T12:44:03.919 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 zonegroup default 2021-09-06T12:44:03.920 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.920 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.920 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.915+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.921 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.916+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.921 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.916+0000 7f31fbab6380 20 rados->read ofs=0 len=0 2021-09-06T12:44:03.921 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.916+0000 7f31fbab6380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:03.921 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.916+0000 7f31fbab6380 20 started sync module instance, tier type = 2021-09-06T12:44:03.922 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.916+0000 7f31fbab6380 20 started zone id=5db7c9f1-7199-4792-8f33-b1c03046b437 (name=default) with tier type = 2021-09-06T12:44:03.926 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=0 2021-09-06T12:44:03.927 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=1 2021-09-06T12:44:03.927 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=2 2021-09-06T12:44:03.927 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=3 2021-09-06T12:44:03.927 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=4 2021-09-06T12:44:03.928 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=5 2021-09-06T12:44:03.928 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.927+0000 7f31fbab6380 20 add_watcher() i=6 2021-09-06T12:44:03.929 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.928+0000 7f31fbab6380 20 add_watcher() i=7 2021-09-06T12:44:03.929 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.928+0000 7f31fbab6380 2 all 8 watchers are set, enabling cache 2021-09-06T12:44:03.929 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.929+0000 7f31bffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:44:03.930 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.930+0000 7f31fbab6380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:44:03.930 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.930+0000 7f31fbab6380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:44:03.933 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.933+0000 7f31fbab6380 5 note: GC not initialized 2021-09-06T12:44:03.934 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.934+0000 7f312effd700 20 reqs_thread_entry: start 2021-09-06T12:44:03.934 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.934+0000 7f312cff9700 20 reqs_thread_entry: start 2021-09-06T12:44:03.935 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.934+0000 7f31fbab6380 20 init_complete bucket index max shards: 11 2021-09-06T12:44:03.935 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.935+0000 7f31fbab6380 10 rgw notify: Started notification manager with: 1 workers 2021-09-06T12:44:03.935 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.935+0000 7f3122ffd700 20 rgw notify: INFO: next queues processing will happen at: Mon Sep 6 12:44:34 2021 2021-09-06T12:44:03.935 INFO:teuthology.orchestra.run.smithi052.stderr: 2021-09-06T12:44:03.936 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.935+0000 7f31227fc700 20 reqs_thread_entry: start 2021-09-06T12:44:03.946 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-09-06T12:44:03.947 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.947+0000 7f31fbab6380 20 remove_watcher() i=0 2021-09-06T12:44:03.947 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.947+0000 7f31fbab6380 2 removed watcher, disabling cache 2021-09-06T12:44:03.948 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.948+0000 7f31fbab6380 20 remove_watcher() i=1 2021-09-06T12:44:03.949 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.950+0000 7f31fbab6380 20 remove_watcher() i=2 2021-09-06T12:44:03.950 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.951+0000 7f31fbab6380 20 remove_watcher() i=3 2021-09-06T12:44:03.951 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.952+0000 7f31fbab6380 20 remove_watcher() i=4 2021-09-06T12:44:03.953 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.953+0000 7f31fbab6380 20 remove_watcher() i=5 2021-09-06T12:44:03.954 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.954+0000 7f31fbab6380 20 remove_watcher() i=6 2021-09-06T12:44:03.955 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.956+0000 7f31fbab6380 20 remove_watcher() i=7 2021-09-06T12:44:03.956 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:03.956+0000 7f31bffe7700 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2021-09-06T12:44:03.960 DEBUG:tasks.util.rgw: json result: [] 2021-09-06T12:44:03.961 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:44:03.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-09-06T12:44:03.962 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-09-06T12:44:04.046 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:04.046 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.052+0000 7fd0b0179380 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-09-06T12:44:04.052 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.052+0000 7fd0b0179380 20 check_secure_mon_conn(): mode 1 is insecure 2021-09-06T12:44:04.055 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.056+0000 7fd03cff9700 20 reqs_thread_entry: start 2021-09-06T12:44:04.056 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.056+0000 7fd0b0179380 20 try_to_resolve_local_entities(): before: realm_id= realm_name= zonegroup_id= zonegroup_name= zone_id= zone_name=default 2021-09-06T12:44:04.056 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.056+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.057 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.057+0000 7fd0b0179380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:04.057 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.057+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.057 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.057+0000 7fd0b0179380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:44:04.068 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 RGWRados::pool_iterate: got zonegroup_info.6179ff01-2837-43c7-8636-bb86386c205f 2021-09-06T12:44:04.069 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 RGWRados::pool_iterate: got zone_info.5db7c9f1-7199-4792-8f33-b1c03046b437 2021-09-06T12:44:04.069 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 RGWRados::pool_iterate: got zone_names.default 2021-09-06T12:44:04.069 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-09-06T12:44:04.069 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.070 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 rados_obj.operate() r=0 bl.length=889 2021-09-06T12:44:04.070 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.069+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.071 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.070+0000 7fd0b0179380 20 rados_obj.operate() r=-2 bl.length=0 2021-09-06T12:44:04.071 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.070+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.071 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.070+0000 7fd0b0179380 20 rados_obj.operate() r=0 bl.length=46 2021-09-06T12:44:04.071 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.070+0000 7fd0b0179380 20 rados->read ofs=0 len=0 2021-09-06T12:44:04.072 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:04.070+0000 7fd0b0179380 20 rados_obj.operate() r=0 bl.length=358 2021-09-06T12:44:04.074 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:04.074 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "5db7c9f1-7199-4792-8f33-b1c03046b437", 2021-09-06T12:44:04.075 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default", 2021-09-06T12:44:04.075 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "default.rgw.meta:root", 2021-09-06T12:44:04.075 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "default.rgw.control", 2021-09-06T12:44:04.075 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "default.rgw.log:gc", 2021-09-06T12:44:04.075 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "default.rgw.log:lc", 2021-09-06T12:44:04.076 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "default.rgw.log", 2021-09-06T12:44:04.076 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "default.rgw.log:intent", 2021-09-06T12:44:04.076 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "default.rgw.log:usage", 2021-09-06T12:44:04.077 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "default.rgw.meta:roles", 2021-09-06T12:44:04.077 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "default.rgw.log:reshard", 2021-09-06T12:44:04.077 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "default.rgw.meta:users.keys", 2021-09-06T12:44:04.077 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "default.rgw.meta:users.email", 2021-09-06T12:44:04.077 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "default.rgw.meta:users.swift", 2021-09-06T12:44:04.078 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "default.rgw.meta:users.uid", 2021-09-06T12:44:04.078 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "default.rgw.otp", 2021-09-06T12:44:04.078 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:44:04.078 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "", 2021-09-06T12:44:04.078 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "" 2021-09-06T12:44:04.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:04.079 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:44:04.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:04.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:44:04.080 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:44:04.080 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "default.rgw.buckets.index", 2021-09-06T12:44:04.080 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:44:04.080 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:44:04.080 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "default.rgw.buckets.data", 2021-09-06T12:44:04.081 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2021-09-06T12:44:04.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:04.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:04.081 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "default.rgw.buckets.non-ec", 2021-09-06T12:44:04.082 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:44:04.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:04.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:04.082 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:04.082 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "", 2021-09-06T12:44:04.083 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "default.rgw.log:notif" 2021-09-06T12:44:04.083 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:04.084 DEBUG:tasks.util.rgw: json result: {'id': '5db7c9f1-7199-4792-8f33-b1c03046b437', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2021-09-06T12:44:04.084 INFO:tasks.rgw:Starting rgw... 2021-09-06T12:44:04.085 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-09-06T12:44:04.085 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T12:44:04.085 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-06T12:44:04.086 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-06T12:44:04.089 INFO:tasks.rgw.c1.client.0:Started 2021-09-06T12:44:04.089 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-09-06T12:44:04.090 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T12:44:04.090 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-06T12:44:04.090 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-06T12:44:04.093 INFO:tasks.rgw.c1.client.1:Started 2021-09-06T12:44:04.093 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-09-06T12:44:04.093 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T12:44:04.094 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-06T12:44:04.094 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-06T12:44:04.096 INFO:tasks.rgw.c2.client.0:Started 2021-09-06T12:44:04.097 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-09-06T12:44:04.097 INFO:tasks.rgw:Using beast as radosgw frontend 2021-09-06T12:44:04.097 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-06T12:44:04.098 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-06T12:44:04.100 INFO:tasks.rgw.c2.client.1:Started 2021-09-06T12:44:04.100 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi041.front.sepia.ceph.com:8000/ 2021-09-06T12:44:04.101 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8000/ 2021-09-06T12:44:04.139 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T12:44:04.140 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:04.140 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:04.140 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi041.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T12:44:05.142 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8000/ 2021-09-06T12:44:05.161 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:05.161 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:05.163 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:05.164 INFO:teuthology.orchestra.run.smithi041.stdout:anonymous 2021-09-06T12:44:05.164 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi041.front.sepia.ceph.com:8001/ 2021-09-06T12:44:05.165 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8001/ 2021-09-06T12:44:05.223 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:05.223 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:05.224 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:05.225 INFO:teuthology.orchestra.run.smithi041.stdout:anonymous 2021-09-06T12:44:05.226 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi052.front.sepia.ceph.com:8000/ 2021-09-06T12:44:05.226 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000/ 2021-09-06T12:44:05.263 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:05.263 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:05.265 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:05.266 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2021-09-06T12:44:05.266 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi052.front.sepia.ceph.com:8001/ 2021-09-06T12:44:05.267 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001/ 2021-09-06T12:44:05.326 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:05.326 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:05.327 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:05.328 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2021-09-06T12:44:05.329 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-09-06T12:44:05.450 INFO:tasks.rgw_multisite:creating realm.. 2021-09-06T12:44:05.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.455 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:05.496 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:05.497 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:05.536 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:05.537 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:05.537 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-realm", 2021-09-06T12:44:05.537 INFO:teuthology.orchestra.run.smithi041.stdout: "current_period": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T12:44:05.537 INFO:teuthology.orchestra.run.smithi041.stdout: "epoch": 1 2021-09-06T12:44:05.538 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:05.539 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-09-06T12:44:05.539 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.540 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:05.621 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:05.621 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:05.633 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:44:05.634+0000 7f79f3a20380 -1 old format 2021-09-06T12:44:05.640 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:05.641 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "test-zonegroup", 2021-09-06T12:44:05.641 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zonegroup", 2021-09-06T12:44:05.641 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "", 2021-09-06T12:44:05.641 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "true", 2021-09-06T12:44:05.641 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:05.642 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:05.642 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:05.642 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T12:44:05.642 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T12:44:05.642 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "", 2021-09-06T12:44:05.643 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [], 2021-09-06T12:44:05.643 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [], 2021-09-06T12:44:05.643 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "", 2021-09-06T12:44:05.643 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:05.644 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T12:44:05.644 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T12:44:05.644 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:05.644 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:05.645 INFO:tasks.rgw_multisite:creating master zone.. 2021-09-06T12:44:05.646 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi041.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi041.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:05.646 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --endpoints http://smithi041.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:05.734 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:05.734 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:05.768 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:05.768 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:05.768 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T12:44:05.768 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-06T12:44:05.769 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-06T12:44:05.769 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-06T12:44:05.769 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-06T12:44:05.769 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-06T12:44:05.770 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-06T12:44:05.770 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-06T12:44:05.770 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-06T12:44:05.770 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-06T12:44:05.771 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-06T12:44:05.771 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-06T12:44:05.771 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-06T12:44:05.771 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-06T12:44:05.772 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-06T12:44:05.772 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:44:05.772 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:05.772 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:05.772 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:05.773 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:44:05.773 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:05.773 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:44:05.773 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:44:05.774 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-06T12:44:05.774 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:44:05.774 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:44:05.774 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "test-zone1.rgw.buckets.data" 2021-09-06T12:44:05.775 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:05.775 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:05.775 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-06T12:44:05.776 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:44:05.776 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:05.776 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:05.776 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:05.776 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:05.777 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-06T12:44:05.777 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:05.778 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-09-06T12:44:06.788 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-09-06T12:44:06.805 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-09-06T12:44:07.791 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-09-06T12:44:07.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:07.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:07.813 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:07.875 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:07.875 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:07.904 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T12:44:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "test-zone1.rgw.meta:root", 2021-09-06T12:44:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "test-zone1.rgw.control", 2021-09-06T12:44:07.906 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "test-zone1.rgw.log:gc", 2021-09-06T12:44:07.906 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "test-zone1.rgw.log:lc", 2021-09-06T12:44:07.906 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "test-zone1.rgw.log", 2021-09-06T12:44:07.906 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "test-zone1.rgw.log:intent", 2021-09-06T12:44:07.907 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "test-zone1.rgw.log:usage", 2021-09-06T12:44:07.907 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "test-zone1.rgw.meta:roles", 2021-09-06T12:44:07.907 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "test-zone1.rgw.log:reshard", 2021-09-06T12:44:07.907 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "test-zone1.rgw.meta:users.keys", 2021-09-06T12:44:07.907 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "test-zone1.rgw.meta:users.email", 2021-09-06T12:44:07.908 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "test-zone1.rgw.meta:users.swift", 2021-09-06T12:44:07.908 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "test-zone1.rgw.meta:users.uid", 2021-09-06T12:44:07.908 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "test-zone1.rgw.otp", 2021-09-06T12:44:07.908 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:44:07.908 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:07.909 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:07.909 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:07.909 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:44:07.909 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:07.910 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:44:07.910 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:44:07.910 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "test-zone1.rgw.buckets.index", 2021-09-06T12:44:07.910 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:44:07.910 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:44:07.911 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "test-zone1.rgw.buckets.data", 2021-09-06T12:44:07.911 INFO:teuthology.orchestra.run.smithi041.stdout: "compression_type": "random" 2021-09-06T12:44:07.911 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:07.911 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:07.912 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "test-zone1.rgw.buckets.non-ec", 2021-09-06T12:44:07.912 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:44:07.912 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:07.912 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:07.912 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:07.913 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "", 2021-09-06T12:44:07.913 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "test-zone1.rgw.log:notif" 2021-09-06T12:44:07.913 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:07.914 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:07.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:07.915 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:07.957 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:07.958 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:07.967 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:44:07.968+0000 7f81736f5380 0 period (b82eb02b-9c46-4fe5-be3a-c10996df6ad0 does not have zone a488f9d2-4d4c-49b4-b839-c5d73c4d316e configured 2021-09-06T12:44:15.898 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:15.898 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:15.899 INFO:teuthology.orchestra.run.smithi041.stdout: "epoch": 1, 2021-09-06T12:44:15.899 INFO:teuthology.orchestra.run.smithi041.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T12:44:15.899 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": [], 2021-09-06T12:44:15.899 INFO:teuthology.orchestra.run.smithi041.stdout: "period_map": { 2021-09-06T12:44:15.900 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:15.900 INFO:teuthology.orchestra.run.smithi041.stdout: "zonegroups": [ 2021-09-06T12:44:15.900 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:15.900 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "test-zonegroup", 2021-09-06T12:44:15.900 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zonegroup", 2021-09-06T12:44:15.901 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "", 2021-09-06T12:44:15.901 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "true", 2021-09-06T12:44:15.901 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:15.901 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:15.902 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:15.902 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T12:44:15.902 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T12:44:15.902 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:15.902 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [ 2021-09-06T12:44:15.903 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:15.903 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:15.903 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T12:44:15.903 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:15.904 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:15.904 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:15.904 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T12:44:15.904 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "false", 2021-09-06T12:44:15.904 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:15.905 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T12:44:15.905 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T12:44:15.905 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T12:44:15.905 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T12:44:15.905 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T12:44:15.906 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.906 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:15.906 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [ 2021-09-06T12:44:15.906 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:15.907 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default-placement", 2021-09-06T12:44:15.907 INFO:teuthology.orchestra.run.smithi041.stdout: "tags": [], 2021-09-06T12:44:15.907 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": [ 2021-09-06T12:44:15.907 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD" 2021-09-06T12:44:15.907 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:44:15.908 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.908 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:15.908 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "default-placement", 2021-09-06T12:44:15.908 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:15.909 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T12:44:15.909 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T12:44:15.909 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.909 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.909 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:15.910 INFO:teuthology.orchestra.run.smithi041.stdout: "short_zone_ids": [ 2021-09-06T12:44:15.910 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:15.910 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:15.910 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 554922684 2021-09-06T12:44:15.910 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.911 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:44:15.911 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:15.911 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:44:15.911 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:15.912 INFO:teuthology.orchestra.run.smithi041.stdout: "period_config": { 2021-09-06T12:44:15.912 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T12:44:15.912 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:15.912 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:15.912 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:15.913 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:15.913 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:15.913 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:15.913 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T12:44:15.914 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:15.914 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:15.914 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:15.914 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:15.915 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:15.915 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:15.915 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:15.915 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:15.915 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_name": "test-realm", 2021-09-06T12:44:15.916 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:44:15.916 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:15.917 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-09-06T12:44:15.918 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T12:44:15.959 INFO:tasks.rgw.c1.client.0.smithi041.stdout:2021-09-06T12:44:15.959+0000 7f8a88757700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 36686) UID: 0 2021-09-06T12:44:15.959 INFO:tasks.rgw.c1.client.0.smithi041.stdout:2021-09-06T12:44:15.959+0000 7f8a995f7440 -1 shutting down 2021-09-06T12:44:20.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-06T12:44:22.019 INFO:tasks.rgw.c1.client.0:Stopped 2021-09-06T12:44:22.020 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-09-06T12:44:22.020 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-09-06T12:44:22.023 INFO:tasks.rgw.c1.client.0:Started 2021-09-06T12:44:22.024 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8000 2021-09-06T12:44:22.046 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:22.046 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:22.046 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi041.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T12:44:22.047 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T12:44:23.055 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8000 2021-09-06T12:44:23.077 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:23.079 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:23.089 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 19454 0 --:--:-- --:--:-- --:--:-- 19454 2021-09-06T12:44:23.089 INFO:teuthology.orchestra.run.smithi041.stdout:anonymous 2021-09-06T12:44:23.090 INFO:tasks.rgw_multisite:creating admin user.. 2021-09-06T12:44:23.090 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.091 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:23.185 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:23.185 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:23.235 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:23.235 INFO:teuthology.orchestra.run.smithi041.stdout: "user_id": "realm-admin", 2021-09-06T12:44:23.235 INFO:teuthology.orchestra.run.smithi041.stdout: "display_name": "Realm Admin", 2021-09-06T12:44:23.236 INFO:teuthology.orchestra.run.smithi041.stdout: "email": "", 2021-09-06T12:44:23.236 INFO:teuthology.orchestra.run.smithi041.stdout: "suspended": 0, 2021-09-06T12:44:23.236 INFO:teuthology.orchestra.run.smithi041.stdout: "max_buckets": 1000, 2021-09-06T12:44:23.236 INFO:teuthology.orchestra.run.smithi041.stdout: "subusers": [], 2021-09-06T12:44:23.237 INFO:teuthology.orchestra.run.smithi041.stdout: "keys": [ 2021-09-06T12:44:23.237 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:23.237 INFO:teuthology.orchestra.run.smithi041.stdout: "user": "realm-admin", 2021-09-06T12:44:23.237 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:23.238 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:23.238 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:23.238 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:23.238 INFO:teuthology.orchestra.run.smithi041.stdout: "swift_keys": [], 2021-09-06T12:44:23.239 INFO:teuthology.orchestra.run.smithi041.stdout: "caps": [], 2021-09-06T12:44:23.239 INFO:teuthology.orchestra.run.smithi041.stdout: "op_mask": "read, write, delete", 2021-09-06T12:44:23.239 INFO:teuthology.orchestra.run.smithi041.stdout: "system": "true", 2021-09-06T12:44:23.239 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "", 2021-09-06T12:44:23.239 INFO:teuthology.orchestra.run.smithi041.stdout: "default_storage_class": "", 2021-09-06T12:44:23.240 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_tags": [], 2021-09-06T12:44:23.240 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T12:44:23.240 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:23.240 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:23.241 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:23.241 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:23.241 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:23.241 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:23.241 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T12:44:23.242 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:23.242 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:23.242 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:23.242 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:23.243 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:23.243 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:23.243 INFO:teuthology.orchestra.run.smithi041.stdout: "temp_url_keys": [], 2021-09-06T12:44:23.243 INFO:teuthology.orchestra.run.smithi041.stdout: "type": "rgw", 2021-09-06T12:44:23.243 INFO:teuthology.orchestra.run.smithi041.stdout: "mfa_ids": [] 2021-09-06T12:44:23.244 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:23.244 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:44:23.249 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-06T12:44:23.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.251 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi041.front.sepia.ceph.com:8000 --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:23.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:23.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:23.397 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:23.397 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:23.398 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-realm", 2021-09-06T12:44:23.398 INFO:teuthology.orchestra.run.smithi052.stdout: "current_period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:23.398 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2 2021-09-06T12:44:23.398 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:23.400 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T12:44:23.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:23.401 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --endpoints http://smithi052.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:23.451 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:23.452 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:23.481 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:23.481+0000 7fb5ce28b380 0 failed reading obj info from .rgw.root:zone_info.a488f9d2-4d4c-49b4-b839-c5d73c4d316e: (2) No such file or directory 2021-09-06T12:44:23.481 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:23.481+0000 7fb5ce28b380 0 WARNING: could not read zone params for zone id=a488f9d2-4d4c-49b4-b839-c5d73c4d316e name=test-zone1 2021-09-06T12:44:23.486 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:23.486 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T12:44:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T12:44:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T12:44:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T12:44:23.487 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T12:44:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T12:44:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T12:44:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T12:44:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T12:44:23.488 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T12:44:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T12:44:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T12:44:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T12:44:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T12:44:23.489 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T12:44:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:44:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:23.490 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:44:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:44:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:44:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T12:44:23.491 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:44:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:44:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data" 2021-09-06T12:44:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:23.492 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T12:44:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:44:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:23.493 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:23.494 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T12:44:23.494 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:23.495 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-09-06T12:44:24.689 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-09-06T12:44:24.709 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-09-06T12:44:25.708 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-09-06T12:44:25.726 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:25.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:25.727 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:25.784 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:25.784 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:25.814 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:25.814 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:25.814 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T12:44:25.814 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T12:44:25.815 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T12:44:25.815 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T12:44:25.815 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T12:44:25.815 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T12:44:25.816 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T12:44:25.816 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T12:44:25.816 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T12:44:25.816 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T12:44:25.816 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T12:44:25.817 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T12:44:25.817 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T12:44:25.817 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T12:44:25.817 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T12:44:25.817 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:44:25.818 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:25.818 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:25.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:25.818 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:44:25.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:25.819 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:44:25.819 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:44:25.820 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T12:44:25.820 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:44:25.820 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:44:25.820 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-06T12:44:25.820 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2021-09-06T12:44:25.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:25.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:25.821 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T12:44:25.821 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:44:25.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:25.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:25.822 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:25.822 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "", 2021-09-06T12:44:25.822 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T12:44:25.823 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:25.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:25.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:25.825 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:25.867 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:25.868 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:25.878 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:25.878+0000 7fd9c7c43380 0 period (588058fc-8215-429d-a24d-21295d48dfdc does not have zone b8f5da72-5870-48fa-9c1e-4ed10a947347 configured 2021-09-06T12:44:26.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-09-06T12:44:33.769 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone a488f9d2-4d4c-49b4-b839-c5d73c4d316e 2021-09-06T12:44:33.817 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2, 2021-09-06T12:44:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T12:44:33.818 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [], 2021-09-06T12:44:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2021-09-06T12:44:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2021-09-06T12:44:33.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2021-09-06T12:44:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2021-09-06T12:44:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2021-09-06T12:44:33.820 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2021-09-06T12:44:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:44:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2021-09-06T12:44:33.821 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2021-09-06T12:44:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2021-09-06T12:44:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.822 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2021-09-06T12:44:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:44:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.823 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:44:33.824 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:44:33.824 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:33.824 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:44:33.824 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T12:44:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T12:44:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T12:44:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:44:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:33.825 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T12:44:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:44:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T12:44:33.826 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:44:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:44:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:33.827 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:44:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T12:44:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T12:44:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T12:44:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:44:33.828 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2021-09-06T12:44:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2021-09-06T12:44:33.829 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2021-09-06T12:44:33.830 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2021-09-06T12:44:33.830 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2021-09-06T12:44:33.830 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:44:33.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2021-09-06T12:44:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2021-09-06T12:44:33.831 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2021-09-06T12:44:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2021-09-06T12:44:33.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 554922684 2021-09-06T12:44:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:33.833 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3217630970 2021-09-06T12:44:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:44:33.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:33.835 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:44:33.835 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:33.835 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2021-09-06T12:44:33.835 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2021-09-06T12:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T12:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T12:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T12:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T12:44:33.836 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T12:44:33.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:33.837 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2021-09-06T12:44:33.837 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T12:44:33.837 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T12:44:33.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T12:44:33.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T12:44:33.838 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T12:44:33.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:33.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:33.839 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:33.839 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2021-09-06T12:44:33.839 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:44:33.839 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:33.841 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-09-06T12:44:33.841 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T12:44:33.883 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:44:33.883+0000 7f6bb95bc700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 36543) UID: 0 2021-09-06T12:44:33.883 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:44:33.883+0000 7f6bca45c440 -1 shutting down 2021-09-06T12:44:37.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-09-06T12:44:39.943 INFO:tasks.rgw.c2.client.0:Stopped 2021-09-06T12:44:39.943 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-09-06T12:44:39.944 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-09-06T12:44:39.947 INFO:tasks.rgw.c2.client.0:Started 2021-09-06T12:44:39.948 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2021-09-06T12:44:39.969 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:39.970 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:39.970 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8000: Connection refused 2021-09-06T12:44:39.971 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T12:44:40.972 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2021-09-06T12:44:40.993 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:40.993 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:40.996 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:40.996 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2021-09-06T12:44:40.996 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T12:44:40.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi041.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:40.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi041.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:40.998 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --endpoints http://smithi041.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:41.052 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:41.053 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:41.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:44:41.087+0000 7fc210c28380 0 failed reading obj info from .rgw.root:zone_info.b8f5da72-5870-48fa-9c1e-4ed10a947347: (2) No such file or directory 2021-09-06T12:44:41.086 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:44:41.087+0000 7fc210c28380 0 WARNING: could not read zone params for zone id=b8f5da72-5870-48fa-9c1e-4ed10a947347 name=test-zone2 2021-09-06T12:44:41.090 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:44:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone3", 2021-09-06T12:44:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-06T12:44:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-06T12:44:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-06T12:44:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-06T12:44:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-06T12:44:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-06T12:44:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-06T12:44:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-06T12:44:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-06T12:44:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-06T12:44:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-06T12:44:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-06T12:44:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-06T12:44:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-06T12:44:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:44:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:44:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:44:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:44:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-06T12:44:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:44:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:44:41.097 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "test-zone3.rgw.buckets.data" 2021-09-06T12:44:41.097 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:41.097 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:41.098 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-06T12:44:41.098 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:44:41.098 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:41.099 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:41.099 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:41.099 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:41.099 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-06T12:44:41.100 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:41.101 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-09-06T12:44:41.815 INFO:teuthology.orchestra.run.smithi041.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-09-06T12:44:41.831 DEBUG:teuthology.orchestra.run.smithi041:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-09-06T12:44:42.831 INFO:teuthology.orchestra.run.smithi041.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-09-06T12:44:42.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:42.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:42.851 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:42.904 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:42.904 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:42.934 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:42.934 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:44:42.934 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone3", 2021-09-06T12:44:42.935 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "test-zone3.rgw.meta:root", 2021-09-06T12:44:42.935 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "test-zone3.rgw.control", 2021-09-06T12:44:42.935 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "test-zone3.rgw.log:gc", 2021-09-06T12:44:42.935 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "test-zone3.rgw.log:lc", 2021-09-06T12:44:42.936 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "test-zone3.rgw.log", 2021-09-06T12:44:42.936 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "test-zone3.rgw.log:intent", 2021-09-06T12:44:42.936 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "test-zone3.rgw.log:usage", 2021-09-06T12:44:42.936 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "test-zone3.rgw.meta:roles", 2021-09-06T12:44:42.936 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "test-zone3.rgw.log:reshard", 2021-09-06T12:44:42.937 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "test-zone3.rgw.meta:users.keys", 2021-09-06T12:44:42.937 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "test-zone3.rgw.meta:users.email", 2021-09-06T12:44:42.937 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "test-zone3.rgw.meta:users.swift", 2021-09-06T12:44:42.937 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "test-zone3.rgw.meta:users.uid", 2021-09-06T12:44:42.937 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "test-zone3.rgw.otp", 2021-09-06T12:44:42.938 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T12:44:42.938 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:42.938 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:42.938 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:42.939 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T12:44:42.939 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:42.939 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T12:44:42.939 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:44:42.939 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "test-zone3.rgw.buckets.index", 2021-09-06T12:44:42.940 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T12:44:42.940 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T12:44:42.940 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "test-zone3.rgw.buckets.data", 2021-09-06T12:44:42.940 INFO:teuthology.orchestra.run.smithi041.stdout: "compression_type": "random" 2021-09-06T12:44:42.940 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:42.941 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:42.941 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "test-zone3.rgw.buckets.non-ec", 2021-09-06T12:44:42.941 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T12:44:42.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:42.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:42.942 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:42.942 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "", 2021-09-06T12:44:42.942 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "test-zone3.rgw.log:notif" 2021-09-06T12:44:42.942 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:42.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:42.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:42.944 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:42.986 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:42.987 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:42.997 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:44:42.998+0000 7fb461241380 0 period (588058fc-8215-429d-a24d-21295d48dfdc does not have zone 4561051f-a4da-4b2a-84e5-f27920586526 configured 2021-09-06T12:44:43.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-09-06T12:44:50.890 INFO:teuthology.orchestra.run.smithi041.stderr:Sending period to new master zone a488f9d2-4d4c-49b4-b839-c5d73c4d316e 2021-09-06T12:44:50.925 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:44:50.925 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:50.925 INFO:teuthology.orchestra.run.smithi041.stdout: "epoch": 3, 2021-09-06T12:44:50.925 INFO:teuthology.orchestra.run.smithi041.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T12:44:50.926 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": [], 2021-09-06T12:44:50.926 INFO:teuthology.orchestra.run.smithi041.stdout: "period_map": { 2021-09-06T12:44:50.926 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:50.926 INFO:teuthology.orchestra.run.smithi041.stdout: "zonegroups": [ 2021-09-06T12:44:50.926 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "test-zonegroup", 2021-09-06T12:44:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zonegroup", 2021-09-06T12:44:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "", 2021-09-06T12:44:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "true", 2021-09-06T12:44:50.927 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:50.928 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:50.928 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.928 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T12:44:50.928 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T12:44:50.929 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:50.929 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [ 2021-09-06T12:44:50.929 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.929 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:44:50.929 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone3", 2021-09-06T12:44:50.930 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:50.930 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8001" 2021-09-06T12:44:50.930 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.930 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T12:44:50.930 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T12:44:50.931 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:50.931 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T12:44:50.931 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T12:44:50.931 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T12:44:50.931 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T12:44:50.932 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T12:44:50.932 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.932 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.932 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:50.932 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T12:44:50.933 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:50.933 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:44:50.933 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.933 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T12:44:50.933 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T12:44:50.934 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:50.934 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T12:44:50.934 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T12:44:50.934 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T12:44:50.934 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T12:44:50.935 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T12:44:50.935 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.935 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.935 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:50.935 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone2", 2021-09-06T12:44:50.936 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T12:44:50.936 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T12:44:50.936 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.936 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T12:44:50.936 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T12:44:50.937 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:44:50.937 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T12:44:50.937 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T12:44:50.937 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T12:44:50.937 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T12:44:50.938 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T12:44:50.938 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.938 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.938 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [ 2021-09-06T12:44:50.938 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.939 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default-placement", 2021-09-06T12:44:50.939 INFO:teuthology.orchestra.run.smithi041.stdout: "tags": [], 2021-09-06T12:44:50.939 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": [ 2021-09-06T12:44:50.939 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD" 2021-09-06T12:44:50.939 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:44:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "default-placement", 2021-09-06T12:44:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T12:44:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T12:44:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:44:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "short_zone_ids": [ 2021-09-06T12:44:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:44:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3257390013 2021-09-06T12:44:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 554922684 2021-09-06T12:44:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:44:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:44:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3217630970 2021-09-06T12:44:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:44:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:44:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:44:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "period_config": { 2021-09-06T12:44:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T12:44:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T12:44:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:44:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:44:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:44:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:44:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:44:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:44:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:44:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_name": "test-realm", 2021-09-06T12:44:50.950 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:44:50.950 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:44:50.950 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-09-06T12:44:50.951 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T12:44:50.989 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:44:50.991+0000 7f79fe61d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 36689) UID: 0 2021-09-06T12:44:50.990 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:44:50.991+0000 7f7a0f4bd440 -1 shutting down 2021-09-06T12:44:54.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-09-06T12:44:57.053 INFO:tasks.rgw.c1.client.1:Stopped 2021-09-06T12:44:57.054 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-09-06T12:44:57.054 DEBUG:teuthology.orchestra.run.smithi041:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-09-06T12:44:57.057 INFO:tasks.rgw.c1.client.1:Started 2021-09-06T12:44:57.058 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8001 2021-09-06T12:44:57.080 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:57.080 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T12:44:57.082 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi041.front.sepia.ceph.com port 8001: Connection refused 2021-09-06T12:44:58.083 DEBUG:teuthology.orchestra.run.smithi041:> curl http://smithi041.front.sepia.ceph.com:8001 2021-09-06T12:44:58.103 INFO:teuthology.orchestra.run.smithi041.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:44:58.103 INFO:teuthology.orchestra.run.smithi041.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:44:58.105 INFO:teuthology.orchestra.run.smithi041.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 104k 0 --:--:-- --:--:-- --:--:-- 104k 2021-09-06T12:44:58.106 INFO:teuthology.orchestra.run.smithi041.stdout:anonymous 2021-09-06T12:44:58.107 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-09-06T12:44:58.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:58.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--url', 'http://smithi041.front.sepia.ceph.com:8000', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:58.108 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi041.front.sepia.ceph.com:8000 --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --url http://smithi041.front.sepia.ceph.com:8000 --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:58.166 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:58.167 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:58.205 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:44:58.207+0000 7f03efe11700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.9b968e6d-c378-4ac6-abc0-926118720ae1.control err=-107 2021-09-06T12:44:58.217 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:58.218 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:58.218 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-realm", 2021-09-06T12:44:58.218 INFO:teuthology.orchestra.run.smithi052.stdout: "current_period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:44:58.219 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 2 2021-09-06T12:44:58.219 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:58.220 INFO:tasks.rgw_multisite:creating zone.. 2021-09-06T12:44:58.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:58.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'IH9RGB4MJICQ8BMT', '--secret', 'ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:58.222 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key IH9RGB4MJICQ8BMT --secret ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu --endpoints http://smithi052.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:58.278 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:58.278 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:44:58.307 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:58.308+0000 7fde799b0380 0 failed reading obj info from .rgw.root:zone_info.4561051f-a4da-4b2a-84e5-f27920586526: (2) No such file or directory 2021-09-06T12:44:58.307 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:58.308+0000 7fde799b0380 0 WARNING: could not read zone params for zone id=4561051f-a4da-4b2a-84e5-f27920586526 name=test-zone3 2021-09-06T12:44:58.308 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:58.309+0000 7fde799b0380 0 failed reading obj info from .rgw.root:zone_info.a488f9d2-4d4c-49b4-b839-c5d73c4d316e: (2) No such file or directory 2021-09-06T12:44:58.308 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:44:58.309+0000 7fde799b0380 0 WARNING: could not read zone params for zone id=a488f9d2-4d4c-49b4-b839-c5d73c4d316e name=test-zone1 2021-09-06T12:44:58.312 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:44:58.312 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T12:44:58.312 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2021-09-06T12:44:58.313 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-06T12:44:58.313 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-06T12:44:58.313 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-06T12:44:58.313 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-06T12:44:58.313 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-06T12:44:58.314 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-06T12:44:58.314 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-06T12:44:58.314 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-06T12:44:58.314 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-06T12:44:58.315 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-06T12:44:58.315 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-06T12:44:58.315 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-06T12:44:58.315 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-06T12:44:58.315 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-06T12:44:58.316 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:44:58.316 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:44:58.316 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:44:58.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:58.317 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:44:58.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:44:58.317 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:44:58.317 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:44:58.317 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-06T12:44:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:44:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:44:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone4.rgw.buckets.data" 2021-09-06T12:44:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:58.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:58.319 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-06T12:44:58.319 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:44:58.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:58.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:44:58.320 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:44:58.320 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_config": { 2021-09-06T12:44:58.320 INFO:teuthology.orchestra.run.smithi052.stdout: "event_retention_days": 7, 2021-09-06T12:44:58.320 INFO:teuthology.orchestra.run.smithi052.stdout: "start_with_full_sync": false 2021-09-06T12:44:58.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:44:58.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:44:58.321 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-06T12:44:58.321 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:44:58.322 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-09-06T12:44:58.886 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-09-06T12:44:58.904 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-09-06T12:44:59.889 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-09-06T12:44:59.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:59.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:44:59.924 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:44:59.988 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:44:59.988 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:00.017 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:00.018 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T12:45:00.018 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2021-09-06T12:45:00.018 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone4.rgw.meta:root", 2021-09-06T12:45:00.018 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone4.rgw.control", 2021-09-06T12:45:00.019 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone4.rgw.log:gc", 2021-09-06T12:45:00.019 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone4.rgw.log:lc", 2021-09-06T12:45:00.019 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone4.rgw.log", 2021-09-06T12:45:00.019 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone4.rgw.log:intent", 2021-09-06T12:45:00.019 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone4.rgw.log:usage", 2021-09-06T12:45:00.020 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone4.rgw.meta:roles", 2021-09-06T12:45:00.020 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone4.rgw.log:reshard", 2021-09-06T12:45:00.020 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone4.rgw.meta:users.keys", 2021-09-06T12:45:00.020 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone4.rgw.meta:users.email", 2021-09-06T12:45:00.020 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone4.rgw.meta:users.swift", 2021-09-06T12:45:00.021 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone4.rgw.meta:users.uid", 2021-09-06T12:45:00.021 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone4.rgw.otp", 2021-09-06T12:45:00.021 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T12:45:00.021 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T12:45:00.021 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T12:45:00.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:00.022 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T12:45:00.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:00.022 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T12:45:00.023 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:00.023 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone4.rgw.buckets.index", 2021-09-06T12:45:00.023 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T12:45:00.023 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T12:45:00.023 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone4.rgw.buckets.data", 2021-09-06T12:45:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2021-09-06T12:45:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone4.rgw.buckets.non-ec", 2021-09-06T12:45:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T12:45:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_config": { 2021-09-06T12:45:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: "event_retention_days": 7, 2021-09-06T12:45:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "start_with_full_sync": false 2021-09-06T12:45:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "", 2021-09-06T12:45:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "test-zone4.rgw.log:notif" 2021-09-06T12:45:00.026 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:00.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:45:00.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:45:00.028 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T12:45:00.071 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:00.071 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:00.081 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:45:00.083+0000 7fbffdae3380 0 period (588058fc-8215-429d-a24d-21295d48dfdc does not have zone 8c2685c1-df19-46c2-8a62-6cb4561f09e2 configured 2021-09-06T12:45:00.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-09-06T12:45:07.969 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone a488f9d2-4d4c-49b4-b839-c5d73c4d316e 2021-09-06T12:45:08.004 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:08.005 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:08.005 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 4, 2021-09-06T12:45:08.005 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T12:45:08.006 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [], 2021-09-06T12:45:08.006 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2021-09-06T12:45:08.006 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:08.006 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2021-09-06T12:45:08.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.007 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2021-09-06T12:45:08.007 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2021-09-06T12:45:08.007 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2021-09-06T12:45:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2021-09-06T12:45:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:45:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:45:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.008 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2021-09-06T12:45:08.009 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2021-09-06T12:45:08.009 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:45:08.009 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2021-09-06T12:45:08.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.010 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:45:08.010 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone3", 2021-09-06T12:45:08.010 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:45:08.010 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8001" 2021-09-06T12:45:08.010 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.011 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:45:08.011 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:45:08.011 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:45:08.011 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:45:08.012 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T12:45:08.012 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T12:45:08.012 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T12:45:08.012 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:45:08.012 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.013 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T12:45:08.013 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2021-09-06T12:45:08.013 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:45:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2021-09-06T12:45:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:45:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:45:08.014 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:45:08.015 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:45:08.015 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "pubsub", 2021-09-06T12:45:08.015 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2021-09-06T12:45:08.015 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [ 2021-09-06T12:45:08.016 INFO:teuthology.orchestra.run.smithi052.stdout: "test-zone1" 2021-09-06T12:45:08.016 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.016 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:45:08.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.017 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:45:08.017 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2021-09-06T12:45:08.017 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:45:08.017 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T12:45:08.017 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.018 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:45:08.018 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:45:08.018 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:45:08.018 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:45:08.019 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T12:45:08.019 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T12:45:08.019 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T12:45:08.019 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:45:08.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.020 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.020 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:45:08.020 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T12:45:08.020 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T12:45:08.020 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T12:45:08.021 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.021 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T12:45:08.021 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T12:45:08.021 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T12:45:08.022 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T12:45:08.022 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T12:45:08.022 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T12:45:08.022 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T12:45:08.023 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T12:45:08.023 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.023 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.023 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2021-09-06T12:45:08.023 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.024 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2021-09-06T12:45:08.024 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2021-09-06T12:45:08.024 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2021-09-06T12:45:08.024 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2021-09-06T12:45:08.024 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:08.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.025 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.025 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2021-09-06T12:45:08.025 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:45:08.026 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2021-09-06T12:45:08.026 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2021-09-06T12:45:08.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.026 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.027 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T12:45:08.027 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2021-09-06T12:45:08.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T12:45:08.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3257390013 2021-09-06T12:45:08.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.029 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T12:45:08.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 2300744410 2021-09-06T12:45:08.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.030 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:45:08.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 554922684 2021-09-06T12:45:08.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:08.031 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T12:45:08.031 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3217630970 2021-09-06T12:45:08.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.032 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:08.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.032 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T12:45:08.033 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T12:45:08.033 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2021-09-06T12:45:08.033 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2021-09-06T12:45:08.033 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T12:45:08.033 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T12:45:08.034 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T12:45:08.034 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T12:45:08.034 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T12:45:08.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.035 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2021-09-06T12:45:08.035 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T12:45:08.035 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T12:45:08.035 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T12:45:08.036 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T12:45:08.036 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T12:45:08.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:08.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:08.036 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T12:45:08.037 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2021-09-06T12:45:08.037 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:08.037 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:08.038 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-09-06T12:45:08.038 INFO:teuthology.orchestra.run:waiting for 300 2021-09-06T12:45:08.070 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:45:08.071+0000 7fb4a073a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 36544) UID: 0 2021-09-06T12:45:08.070 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:45:08.071+0000 7fb4b15da440 -1 shutting down 2021-09-06T12:45:12.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-09-06T12:45:14.140 INFO:tasks.rgw.c2.client.1:Stopped 2021-09-06T12:45:14.141 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-09-06T12:45:14.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-09-06T12:45:14.145 INFO:tasks.rgw.c2.client.1:Started 2021-09-06T12:45:14.145 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001 2021-09-06T12:45:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-09-06T12:45:14.168 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:45:14.169 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:45:14.169 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8001: Connection refused 2021-09-06T12:45:15.170 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8001 2021-09-06T12:45:15.191 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-09-06T12:45:15.191 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-09-06T12:45:15.192 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 130k 0 --:--:-- --:--:-- --:--:-- 130k 2021-09-06T12:45:15.193 INFO:teuthology.orchestra.run.smithi052.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-0061360d5b-10ed-test-zone4","HostId":"10ed-test-zone4-test-zonegroup"} 2021-09-06T12:45:15.194 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-09-06T12:45:15.195 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-09-06T12:45:15.206 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa/../src/test/rgw/rgw_multi 2021-09-06T12:45:15.221 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-09-06T12:45:15.222 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:45:15.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T12:45:15.223 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T12:45:15.277 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:15.278 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:15.335 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:45:15.336 INFO:teuthology.orchestra.run.smithi041.stdout: "user_id": "rgw-multisite-test-user", 2021-09-06T12:45:15.336 INFO:teuthology.orchestra.run.smithi041.stdout: "display_name": "Multisite Test User", 2021-09-06T12:45:15.336 INFO:teuthology.orchestra.run.smithi041.stdout: "email": "", 2021-09-06T12:45:15.336 INFO:teuthology.orchestra.run.smithi041.stdout: "suspended": 0, 2021-09-06T12:45:15.336 INFO:teuthology.orchestra.run.smithi041.stdout: "max_buckets": 1000, 2021-09-06T12:45:15.337 INFO:teuthology.orchestra.run.smithi041.stdout: "subusers": [], 2021-09-06T12:45:15.337 INFO:teuthology.orchestra.run.smithi041.stdout: "keys": [ 2021-09-06T12:45:15.337 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:45:15.337 INFO:teuthology.orchestra.run.smithi041.stdout: "user": "rgw-multisite-test-user", 2021-09-06T12:45:15.338 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "7FQFPIGJRWN4507ZDYB3", 2021-09-06T12:45:15.338 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "JhmWCa9nLMDpwBtRo21wRW27vBx0mJLyFYXMJiLx" 2021-09-06T12:45:15.338 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:45:15.338 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T12:45:15.338 INFO:teuthology.orchestra.run.smithi041.stdout: "swift_keys": [], 2021-09-06T12:45:15.339 INFO:teuthology.orchestra.run.smithi041.stdout: "caps": [], 2021-09-06T12:45:15.339 INFO:teuthology.orchestra.run.smithi041.stdout: "op_mask": "read, write, delete", 2021-09-06T12:45:15.339 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "", 2021-09-06T12:45:15.339 INFO:teuthology.orchestra.run.smithi041.stdout: "default_storage_class": "", 2021-09-06T12:45:15.340 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_tags": [], 2021-09-06T12:45:15.340 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T12:45:15.340 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:45:15.340 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:45:15.340 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:45:15.341 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:45:15.341 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:45:15.341 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:45:15.341 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T12:45:15.341 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T12:45:15.342 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T12:45:15.342 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T12:45:15.342 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T12:45:15.343 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T12:45:15.343 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:45:15.343 INFO:teuthology.orchestra.run.smithi041.stdout: "temp_url_keys": [], 2021-09-06T12:45:15.344 INFO:teuthology.orchestra.run.smithi041.stdout: "type": "rgw", 2021-09-06T12:45:15.344 INFO:teuthology.orchestra.run.smithi041.stdout: "mfa_ids": [] 2021-09-06T12:45:15.345 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:45:15.345 INFO:teuthology.orchestra.run.smithi041.stdout: 2021-09-06T12:45:15.351 INFO:rgw_multi.tests:meta checkpoint 2021-09-06T12:45:15.352 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:15.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:15.353 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:15.395 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:15.396 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:15.440 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:45:15.455 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:45:15.456 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:45:15.456 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:45:15.456 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:45:15.457 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:45:15.457 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:45:15.457 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:45:15.457 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:45:15.458 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:45:15.458 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:45:15.458 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:45:15.459 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:45:15.459 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:45:15.459 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:45:15.459 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:45:15.460 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:45:15.460 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:45:15.460 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:45:15.461 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:45:15.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:15.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:15.463 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:15.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:15.512 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:15.576 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:15.577 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:15.578 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:15.579 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:15.580 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:15.581 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:15.582 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:15.583 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:15.584 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:15.585 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:15.586 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:15.587 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:15.587 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:15.588 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:15.588 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:15.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:15.589 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:15.589 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:15.589 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:15.590 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:15.590 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:15.590 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:15.590 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:15.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:15.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:15.592 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:15.592 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:15.592 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:15.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:15.592 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:15.594 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:17.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-09-06T12:45:20.594 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:20.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:20.596 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:20.647 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:20.648 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:20.715 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:20.715 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:20.716 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:20.718 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:20.719 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:20.730 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:20.731 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:25.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:25.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:25.734 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:25.782 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:25.782 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:25.850 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:25.851 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:25.852 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:25.853 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:25.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:25.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:25.856 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:25.857 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:25.858 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:25.859 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.860 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:25.861 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:25.862 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.863 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:25.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:25.865 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:25.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:30.867 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:30.867 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:30.868 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:30.918 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:30.919 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:30.986 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:30.986 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:30.987 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:30.987 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:30.987 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:30.987 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:30.988 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:30.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:30.988 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:30.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:30.988 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:30.989 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:30.989 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:30.989 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:30.989 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:30.990 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:30.990 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:30.990 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:30.990 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:30.990 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:30.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:30.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:30.991 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:30.991 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:30.992 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:30.992 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:30.992 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:30.992 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:30.992 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:30.993 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:30.993 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:30.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:30.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:30.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:30.994 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:30.994 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:30.994 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:30.994 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:30.994 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:30.995 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:30.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:30.995 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:30.995 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:30.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:30.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:30.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:30.996 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:30.996 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:30.997 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:30.997 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:30.997 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:30.997 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:30.997 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:30.998 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:30.998 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:30.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:30.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:30.999 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:30.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:30.999 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:30.999 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:30.999 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:31.000 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:31.000 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:31.001 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:36.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:36.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:36.003 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:36.053 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:36.054 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:36.124 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:36.124 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:36.124 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:36.125 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:36.125 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:36.125 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:36.125 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:36.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:36.126 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:36.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:36.126 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:36.126 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:36.126 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:36.127 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:36.127 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:36.127 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:36.127 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:36.128 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:36.128 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:36.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:36.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:36.129 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:36.129 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:36.129 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:36.129 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:36.130 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:36.130 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:36.130 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:36.130 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:36.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:36.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:36.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:36.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:36.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:36.132 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:36.132 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:36.132 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:36.132 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:36.133 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:36.133 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:36.133 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:36.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:36.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:36.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:36.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:36.134 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:36.134 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:36.135 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:36.135 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:36.135 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:36.135 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:36.135 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:36.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.136 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:36.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:36.136 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:36.137 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:36.137 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:36.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:36.138 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:36.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:41.140 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:41.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:41.141 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:41.192 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:41.192 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:41.258 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:41.259 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:41.259 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:41.259 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:41.259 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:41.260 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:41.260 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:41.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:41.260 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:41.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:41.261 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:41.261 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:41.261 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:41.261 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:41.262 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:41.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:41.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:41.262 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:41.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:41.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:41.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:41.263 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:41.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:41.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:41.264 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:41.264 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:41.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:41.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:41.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:41.265 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:41.265 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:41.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:41.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:41.266 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:41.267 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:41.267 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:41.267 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:41.267 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:41.267 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:41.268 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:41.268 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:41.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:41.269 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:41.270 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:41.270 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:41.270 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:41.270 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:41.270 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:41.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.271 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:41.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:41.271 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:41.272 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:41.272 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:41.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:41.272 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:41.273 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:46.274 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:46.275 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:46.275 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:46.327 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:46.327 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:46.394 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:46.394 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:46.395 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:46.395 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:46.395 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:46.395 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:46.396 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:46.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:46.396 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:46.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:46.396 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:46.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:46.397 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:46.397 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:46.397 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:46.398 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:46.398 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:46.398 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:46.398 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:46.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:46.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:46.399 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:46.400 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:46.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:46.400 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:46.400 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:46.401 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:46.401 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:46.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:46.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:46.402 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:46.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:46.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:46.403 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:46.404 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:46.404 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:46.404 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:46.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:46.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:46.405 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:46.405 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:46.406 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:46.406 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:46.406 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:46.406 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:46.407 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:46.407 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:46.407 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:46.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.408 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:46.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:46.408 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:46.409 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:46.409 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:46.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:46.409 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:46.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:51.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:51.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:51.413 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:51.463 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:51.464 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:51.533 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:51.534 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:51.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:51.536 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:51.537 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:51.538 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:51.539 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:51.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:51.541 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:51.542 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:51.542 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:51.542 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:51.542 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:51.542 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:51.543 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:51.543 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:51.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:51.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:51.544 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:51.544 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:51.544 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:51.544 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:51.545 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:51.545 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:51.545 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:51.545 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:51.545 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:51.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.546 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:51.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:51.547 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:51.547 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:51.547 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:51.547 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:51.547 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:51.548 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:45:56.550 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:56.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:45:56.551 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:45:56.601 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:45:56.601 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:45:56.670 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:45:56.670 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:45:56.670 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:45:56.671 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:45:56.671 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:45:56.671 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:45:56.671 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:56.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:56.672 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:45:56.672 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:56.672 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:45:56.672 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:56.673 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:56.673 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:56.673 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:56.673 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:45:56.674 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:56.674 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:56.674 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:56.674 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:56.675 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:56.675 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:45:56.675 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:56.675 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:56.675 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:56.676 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:56.676 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:56.676 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:56.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:56.677 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:56.677 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.677 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:56.677 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:56.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:45:56.678 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:56.678 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:56.678 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:56.678 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:56.678 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:56.679 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:56.679 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:56.679 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:56.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:56.680 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:45:56.680 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:45:56.680 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:45:56.680 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:45:56.681 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:45:56.681 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:45:56.681 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:45:56.681 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:45:56.682 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:45:56.682 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:45:56.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.682 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:45:56.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:45:56.683 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:45:56.683 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:45:56.683 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:45:56.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:45:56.684 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:45:56.685 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:01.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:01.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:01.687 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:01.742 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:01.743 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:01.811 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:01.812 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:01.812 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:01.812 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:01.813 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:01.813 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:01.813 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:01.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:01.814 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:01.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:01.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:01.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:01.815 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:01.815 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:01.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:01.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:01.816 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:01.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:01.816 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:01.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:01.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:01.817 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:01.817 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:01.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:01.818 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:01.818 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:01.818 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:01.818 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:01.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:01.819 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:01.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:01.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:01.820 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:01.820 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:01.820 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:01.820 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:01.820 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:01.821 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:01.821 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:01.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:01.821 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:01.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:01.822 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:01.822 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:01.823 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:01.823 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:01.823 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:01.823 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:01.823 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:01.824 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:01.824 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:01.824 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:01.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.825 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:01.825 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:01.825 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:01.825 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:01.826 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:01.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:01.826 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:01.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:06.828 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:06.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:06.829 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:06.880 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:06.881 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:06.951 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:06.952 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:06.953 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:06.954 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:06.955 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:06.957 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:06.958 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:06.959 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:06.960 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:06.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:06.962 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:06.963 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:06.964 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:06.965 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:06.966 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:06.967 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:06.968 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:11.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:11.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:11.970 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:12.017 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:12.017 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:12.082 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:12.083 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:12.084 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:12.085 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:12.086 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:12.087 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:12.088 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:12.089 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:12.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:12.091 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:12.093 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:12.094 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.095 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:12.096 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:12.097 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:12.097 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:12.098 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:17.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:17.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:17.100 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:17.151 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:17.152 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:17.221 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:17.221 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:17.221 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:17.222 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:17.222 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:17.222 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:17.223 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:17.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:17.223 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:17.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:17.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:17.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:17.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:17.224 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:17.224 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:17.225 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:17.225 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:17.225 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:17.225 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:17.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:17.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:17.226 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:17.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:17.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:17.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:17.227 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:17.227 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:17.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:17.229 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:17.229 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:17.230 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:17.230 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:17.230 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:17.231 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:17.231 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:17.231 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:17.231 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:17.232 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:17.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:17.232 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:17.232 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:17.233 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:17.233 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:17.233 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:17.233 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:17.233 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:17.234 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:17.234 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:17.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.235 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:17.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:17.235 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:17.235 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:17.236 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:17.236 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:17.236 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:17.237 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:22.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:22.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:22.240 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:22.292 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:22.293 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:22.360 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:22.361 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:22.362 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:22.362 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:22.362 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:22.362 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:22.363 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:22.363 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:22.363 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:22.363 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:22.363 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:22.364 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:22.364 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:22.364 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:22.364 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:22.365 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:22.366 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:22.366 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:22.366 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:22.366 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:22.366 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:22.367 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:22.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:22.367 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:22.367 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:22.367 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:22.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:22.368 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:22.368 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:22.369 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:22.369 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:22.369 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:22.369 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:22.369 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:22.370 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:22.370 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:22.370 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.370 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:22.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:22.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:22.371 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:22.371 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:22.371 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:22.372 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:22.372 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:22.372 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:22.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:22.373 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:22.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.374 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:22.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:22.374 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:22.374 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:22.375 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:22.375 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:22.375 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:22.377 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:27.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:27.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:27.379 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:27.430 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:27.430 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:27.499 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:27.499 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:27.500 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:27.501 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:27.502 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:27.504 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:27.505 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:27.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:27.507 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:27.508 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:27.510 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:27.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:27.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:27.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:27.513 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:27.515 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:32.516 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:32.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:32.517 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:32.570 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:32.571 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:32.638 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:32.638 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:32.639 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:32.639 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:32.639 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:32.639 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:32.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:32.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:32.640 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:32.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:32.641 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:32.642 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:32.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:32.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:32.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:32.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:32.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:32.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:32.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:32.644 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:32.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:32.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:32.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:32.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:32.646 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:32.647 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:32.647 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:32.647 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:32.647 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:32.648 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:32.648 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:32.648 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:32.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:32.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:32.649 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:32.649 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:32.649 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:32.650 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:32.650 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:32.650 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:32.650 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:32.651 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:32.651 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:32.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.651 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:32.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:32.652 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:32.652 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:32.652 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:32.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:32.653 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:32.654 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:37.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:37.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:37.658 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:37.711 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:37.712 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:37.785 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:37.786 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:37.787 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:37.787 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:37.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:37.788 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:37.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:37.788 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:37.788 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:37.789 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:37.789 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:37.789 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:37.789 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:37.790 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:37.790 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:37.791 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:37.791 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:37.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:37.792 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:37.793 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:37.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:37.794 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:37.795 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:37.795 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:37.795 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:37.796 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:37.796 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:37.797 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.797 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:37.797 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:37.798 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:37.798 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:37.798 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:37.799 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:37.799 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:37.799 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:37.800 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:37.800 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:37.800 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:37.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:37.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:37.801 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:37.801 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:37.802 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:37.802 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:37.802 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:37.802 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:37.803 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:37.803 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:37.803 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:37.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.803 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.804 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:37.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:37.804 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:37.804 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:37.805 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:37.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:37.805 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:37.806 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:42.807 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:42.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:42.808 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:42.859 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:42.859 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:42.927 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:42.928 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:42.928 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:42.928 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:42.928 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:42.929 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:42.929 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:42.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:42.929 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:42.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:42.930 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:42.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:42.930 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:42.930 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:42.931 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:42.931 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:42.931 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:42.931 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:42.932 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:42.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:42.932 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:42.932 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:42.933 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:42.933 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:42.933 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:42.933 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:42.934 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:42.934 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:42.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:42.934 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:42.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:42.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:42.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:42.935 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:42.936 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:42.936 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:42.936 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:42.936 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:42.937 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:42.937 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:42.937 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:42.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:42.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:42.938 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:42.938 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:42.938 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:42.939 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:42.939 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:42.939 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:42.939 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:42.940 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:42.940 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:42.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.940 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:42.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:42.941 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:42.941 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:42.941 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:42.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:42.942 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:42.943 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:47.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:47.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:47.945 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:47.994 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:47.994 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:48.062 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:48.063 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:48.063 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:48.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:48.064 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:48.064 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:48.064 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:48.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:48.064 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:48.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:48.065 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:48.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:48.065 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:48.066 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:48.066 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:48.066 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:48.066 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:48.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:48.067 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:48.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:48.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:48.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:48.068 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:48.069 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:48.069 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:48.069 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:48.070 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:48.070 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:48.070 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:48.070 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:48.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:48.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:48.071 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:48.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:48.072 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:48.072 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:48.073 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:48.073 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:48.073 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:48.074 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:48.074 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:48.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:48.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:48.075 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:48.075 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:48.076 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:48.076 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:48.076 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:48.076 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:48.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:48.077 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:48.077 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:48.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.078 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:48.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:48.078 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:48.078 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:48.078 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:48.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:48.079 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:48.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:53.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:53.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:53.083 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:53.134 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:53.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:53.203 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:53.204 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:53.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:53.206 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:53.207 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:53.208 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:53.209 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:53.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:53.211 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:53.212 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:53.213 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:53.214 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:53.215 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:53.216 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:53.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:53.217 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:53.218 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:46:58.219 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:58.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:46:58.220 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:46:58.270 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:46:58.270 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:46:58.339 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:46:58.339 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:46:58.340 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:46:58.340 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:46:58.340 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:46:58.340 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:46:58.340 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:58.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:58.341 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:46:58.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:58.341 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:46:58.342 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:58.342 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:58.342 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:58.342 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:58.342 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:46:58.343 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:58.343 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:58.343 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:58.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:58.344 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:58.344 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:46:58.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:58.344 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:58.345 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:58.345 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:58.345 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:58.345 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:58.345 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:58.346 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:58.346 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:58.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:58.346 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:46:58.347 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:58.347 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:58.347 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:58.347 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:58.348 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:58.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:58.348 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:58.348 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:58.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:58.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:46:58.349 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:46:58.349 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:46:58.349 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:46:58.350 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:46:58.350 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:46:58.350 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:46:58.350 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:46:58.351 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:46:58.351 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:46:58.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.351 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:46:58.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:46:58.352 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:46:58.352 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:46:58.352 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:46:58.352 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:46:58.353 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:46:58.354 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:47:03.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:03.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:03.356 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:03.406 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:03.407 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:03.476 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:03.476 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:03.477 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:03.478 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:03.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:03.479 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:03.479 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:03.479 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:03.480 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:03.480 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:03.480 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:03.480 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:03.480 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:03.481 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:03.481 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:03.481 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:03.481 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:03.482 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:03.482 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:03.482 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:03.482 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:03.483 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:03.483 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:03.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.483 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:03.483 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:03.484 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:03.485 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:03.486 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:03.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:03.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:03.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:03.487 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:03.488 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:03.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:03.488 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:03.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.489 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:03.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:03.489 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:03.489 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:03.490 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:03.490 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:03.490 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:03.491 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:47:08.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:08.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:08.494 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:08.544 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:08.544 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:08.612 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:08.612 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:08.613 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:08.614 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:08.615 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:08.616 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:08.617 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:08.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:08.619 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:08.620 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:08.621 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:08.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:08.623 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:08.624 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:08.625 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:08.626 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:08.626 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:08.627 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:08.628 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:08.630 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:08.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:08.631 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:08.632 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1630932315.333383_35.1 target= 2021-09-06T12:47:13.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.634 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:13.685 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:13.765 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:13.766 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:13.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:13.768 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:13.769 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:13.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:13.771 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.772 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:13.773 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:13.774 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:13.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:13.776 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:13.777 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.778 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:13.779 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:13.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:13.780 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:13.781 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:13.781 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:13.782 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.783 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:13.839 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:13.840 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:13.906 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:13.906 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:13.907 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:13.907 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:13.907 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:13.907 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:13.908 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:13.908 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:13.908 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:13.908 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:13.908 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:13.909 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:13.909 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:13.909 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:13.909 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:13.910 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:13.910 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:13.910 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:13.910 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:13.911 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.911 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:13.911 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:13.911 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:13.912 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:13.912 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:13.912 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:13.912 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:13.912 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:13.913 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:13.913 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.913 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:13.913 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.914 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:13.914 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:13.914 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:13.914 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:13.914 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:13.915 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:13.915 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:13.915 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:13.915 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:13.916 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.916 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:13.916 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.916 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:13.916 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:13.917 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:13.917 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:13.917 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:13.917 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:13.917 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:13.918 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:13.918 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:13.918 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:13.918 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:13.919 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.919 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.919 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:13.919 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:13.919 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:13.920 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:13.920 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:13.920 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:13.920 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:13.922 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:13.922 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:13.923 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:13.923 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:13.971 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:13.971 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:14.032 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.043 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.044 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.044 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.044 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.044 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.045 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.045 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:14.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.046 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:14.046 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:14.046 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:14.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.047 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:14.048 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:14.048 INFO:rgw_multi.tests:meta checkpoint 2021-09-06T12:47:14.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.050 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:14.094 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:14.094 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:14.140 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:14.155 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:14.155 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.156 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:14.156 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:14.156 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.156 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.157 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.157 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:14.157 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.158 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.158 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.158 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:14.158 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.159 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.159 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.159 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:14.159 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.159 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:14.161 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:14.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.161 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.162 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:14.206 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:14.206 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:14.267 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:14.267 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:14.268 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:14.269 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.270 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:14.271 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:14.272 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.273 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.274 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:14.275 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.276 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.277 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.278 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.279 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.280 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:14.281 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:14.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.282 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:14.283 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:14.283 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:14.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.285 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:14.328 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:14.328 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:14.389 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:14.390 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:14.390 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:14.390 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:14.390 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:14.391 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:14.391 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:14.391 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.391 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:14.392 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.392 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:14.392 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:14.392 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:14.392 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:14.393 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:14.393 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:14.393 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:14.393 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:14.394 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:14.394 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.394 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.394 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.395 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:14.395 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:14.395 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:14.395 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.395 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:14.396 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:14.396 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:14.396 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.397 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:14.397 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.397 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.398 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.398 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:14.398 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:14.398 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:14.399 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.399 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:14.399 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:14.399 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:14.399 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.400 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:14.400 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.400 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.400 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:14.400 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:14.401 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:14.401 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:14.401 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:14.401 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:14.401 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:14.402 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:14.402 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.402 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:14.402 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.403 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.403 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:14.403 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:14.403 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:14.404 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:14.404 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:14.404 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:14.404 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:14.405 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:14.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:14.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:14.407 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:14.450 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:14.451 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:14.514 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:14.514 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:14.514 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:14.514 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:14.515 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:14.515 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:14.515 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.516 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:14.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.516 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:14.516 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.516 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.517 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:14.517 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.517 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:14.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.518 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:14.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.519 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:14.519 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.519 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.519 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.520 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.520 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.520 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.520 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.520 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.521 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.521 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:14.521 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.522 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.522 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.522 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.522 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.523 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.523 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.523 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.523 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:14.524 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:14.524 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:14.525 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:14.525 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:14.525 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:14.525 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:14.525 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:14.526 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:14.526 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:14.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.527 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:14.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:14.527 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:14.527 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:14.528 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:14.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:14.528 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:14.529 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:14.530 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/git.ceph.com_ceph-c_0eb1a7943dd70e2a0b3086ea680284137a187e73/qa/../src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-09-06T12:47:14.562 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-1 2021-09-06T12:47:17.709 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-2 2021-09-06T12:47:21.807 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-3 2021-09-06T12:47:21.863 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:21.863 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:21.864 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:21.907 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:21.907 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:21.952 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:21.966 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:21.967 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:21.967 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:21.967 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:21.967 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:21.968 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:21.968 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:21.968 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:21.968 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:21.969 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:21.969 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932441.843766_81.1", 2021-09-06T12:47:21.969 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:21.843766Z" 2021-09-06T12:47:21.969 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:21.969 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:21.970 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:21.970 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:21.970 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:21.970 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:21.971 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:21.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:21.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:21.973 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:22.016 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:22.017 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:22.079 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:22.079 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:22.080 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:22.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:22.082 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:22.084 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:22.085 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:22.087 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932437.687864_70.1", 2021-09-06T12:47:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:22.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:17.687864Z", 2021-09-06T12:47:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:22.089 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:22.090 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:22.091 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.092 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:22.093 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:22.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:22.094 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:22.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630932441.843766_81.1 target=1_1630932437.687864_70.1 2021-09-06T12:47:27.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.097 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.097 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:27.148 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:27.148 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:27.221 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:27.221 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:27.221 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:27.222 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:27.222 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:27.222 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:27.222 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.223 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:27.223 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.223 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:27.223 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.224 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:27.224 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.224 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:27.224 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.225 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:27.225 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.225 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.225 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.226 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:27.226 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.226 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.227 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.227 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.227 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.227 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.228 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.228 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.228 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.228 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.228 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:27.229 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.229 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.229 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932441.843766_81.1", 2021-09-06T12:47:27.229 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.229 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.230 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.230 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:21.843766Z", 2021-09-06T12:47:27.230 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.230 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.231 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.232 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.232 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.232 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.232 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.232 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:27.234 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:27.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.234 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:27.235 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:27.235 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:27.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.237 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:27.288 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:27.289 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:27.356 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:27.356 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:27.356 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:27.357 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:27.357 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:27.357 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:27.357 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:27.358 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.358 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:27.358 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.358 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:27.359 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:27.359 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:27.359 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:27.359 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:27.360 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:27.360 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:27.360 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:27.360 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:27.360 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.361 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.361 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.361 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:27.361 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:27.361 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:27.362 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:27.362 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:27.362 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:27.362 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:27.363 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:27.363 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.363 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.363 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.364 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:27.364 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:27.364 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:27.364 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932441.843766_81.1", 2021-09-06T12:47:27.364 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:27.365 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:27.365 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:27.365 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:21.843766Z", 2021-09-06T12:47:27.365 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:27.366 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.366 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.366 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.366 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:27.366 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:27.367 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:27.367 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:27.367 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:27.367 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:27.367 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:27.368 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.368 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:27.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.368 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:27.369 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.369 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:27.369 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:27.369 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:27.370 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.370 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:27.371 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:27.372 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:27.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.373 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:27.419 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:27.420 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:27.483 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:27.483 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.485 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:27.486 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.488 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:27.490 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932441.843766_81.1", 2021-09-06T12:47:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.491 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:21.843766Z", 2021-09-06T12:47:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.493 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.494 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:27.495 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:27.496 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:27.497 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.497 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:27.498 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:27.523 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-09-06T12:47:27.524 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-4 2021-09-06T12:47:27.545 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-5 2021-09-06T12:47:27.575 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-6 2021-09-06T12:47:27.606 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.607 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:27.652 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:27.652 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:27.704 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:27.720 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:27.721 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.721 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:27.721 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:27.722 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.722 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.722 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:27.722 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:27.723 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.723 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.723 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:27.723 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:27.593883Z" 2021-09-06T12:47:27.724 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:27.724 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:27.724 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:27.724 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:27.725 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:27.725 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:27.726 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:27.727 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.727 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:27.728 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:27.779 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:27.779 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:27.843 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:27.843 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:27.844 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:27.844 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:27.844 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:27.844 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:27.844 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.845 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:27.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.845 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:27.845 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.846 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.846 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:27.846 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.846 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:27.846 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.847 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:27.847 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.848 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:27.848 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.848 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.849 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.849 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.849 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.849 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.850 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.850 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.850 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.851 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.851 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:27.851 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.851 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.852 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932447.534333_82.1", 2021-09-06T12:47:27.852 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.852 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.852 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.852 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:27.534333Z", 2021-09-06T12:47:27.853 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.853 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.853 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.853 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:27.854 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:27.854 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:27.854 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:27.854 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:27.855 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:27.855 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:27.855 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:27.855 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:27.856 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:27.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.856 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:27.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:27.857 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:27.857 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:27.857 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:27.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:27.857 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:27.859 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630932447.593883_93.1 target=1_1630932447.534333_82.1 2021-09-06T12:47:32.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:32.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:32.861 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:32.911 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:32.912 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:32.980 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:32.980 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:32.980 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:32.981 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:32.981 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:32.981 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:32.981 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:32.982 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:32.982 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:32.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:32.982 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:32.982 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:32.983 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:32.983 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:32.983 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:32.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:32.984 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:32.984 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:32.984 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:32.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:32.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:32.985 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:32.985 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:32.985 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:32.985 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:32.986 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:32.986 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:32.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:32.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:32.987 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:32.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:32.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:32.987 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:32.988 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:32.988 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:32.988 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:32.988 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:32.989 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:32.989 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:32.989 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:32.989 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:32.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:32.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:32.990 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:32.990 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:32.991 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:32.991 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:32.991 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:32.991 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:32.991 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:32.992 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:32.992 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:32.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.992 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:32.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:32.993 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:32.993 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:32.993 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:32.994 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:32.994 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:32.995 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:32.996 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:32.996 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:32.997 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:32.997 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.047 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.047 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:33.116 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:33.117 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:33.117 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:33.117 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:33.117 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:33.118 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:33.118 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.118 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.118 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:33.119 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.119 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:33.119 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.119 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.120 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:33.120 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.120 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:33.120 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.120 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:33.121 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.121 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.121 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.121 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.122 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:33.122 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.122 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.122 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.122 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.123 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.123 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.123 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.123 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.124 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.124 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.124 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.124 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:33.125 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.125 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.125 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:33.125 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.126 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.126 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.126 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:33.126 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.126 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.127 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.127 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.127 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:33.127 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.128 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.128 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.128 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.129 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.129 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.129 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.129 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.130 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.130 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.130 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:33.130 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.130 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:33.131 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:33.131 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:33.131 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.131 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:33.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:33.133 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:33.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.135 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.180 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:33.244 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:33.245 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:33.245 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:33.245 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:33.245 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:33.246 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:33.246 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.246 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.246 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:33.246 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.247 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:33.247 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.247 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.247 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:33.248 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.248 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:33.248 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.248 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:33.249 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.249 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.250 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:33.250 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.250 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.250 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:33.250 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.251 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.251 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.251 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.251 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.252 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:33.252 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.253 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.253 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:33.253 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.253 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.253 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.254 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:33.254 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.255 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:33.255 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.255 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.255 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:33.255 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.256 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.256 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.256 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.256 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.257 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:33.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.258 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:33.258 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:33.258 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:33.258 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.258 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:33.259 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:33.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.517 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.561 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.561 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:33.607 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:33.622 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:33.623 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.623 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:33.623 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:33.623 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.624 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.624 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.624 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:33.624 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.624 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.625 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:33.625 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:27.593883Z" 2021-09-06T12:47:33.625 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.625 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.625 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.626 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:33.626 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.626 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:33.627 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:33.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.629 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.674 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.675 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:33.744 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:33.745 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:33.745 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:33.745 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:33.745 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:33.746 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:33.746 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.746 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.746 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:33.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.747 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:33.747 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.747 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.747 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:33.748 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.749 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.749 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:33.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.750 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.750 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:33.750 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.751 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.751 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.751 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:33.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.753 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.753 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:33.753 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.753 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.753 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.754 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:33.754 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.755 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:33.755 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:33.755 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:33.755 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:33.756 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:33.756 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:33.756 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:33.756 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:33.756 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.757 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:33.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.757 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:33.757 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:33.758 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:33.758 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:33.758 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:33.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:33.759 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:33.760 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:33.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:33.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.762 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.811 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.812 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:33.878 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:33.879 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:33.879 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:33.879 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:33.880 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:33.880 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:33.880 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.880 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.881 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:33.881 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.881 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:33.881 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.882 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.882 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:33.882 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.882 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:33.882 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.883 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:33.883 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.883 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.884 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.884 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.884 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:33.884 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.884 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.885 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.885 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.885 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.885 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.886 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.886 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.886 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.886 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.887 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.887 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:33.887 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.887 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.887 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:33.888 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.888 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.888 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.888 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:33.889 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.889 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.889 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.889 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:33.889 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:33.890 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:33.890 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:33.890 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:33.890 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:33.891 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:33.891 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:33.891 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:33.891 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:33.891 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.892 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.892 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:33.892 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:33.893 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:33.893 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:33.893 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:33.893 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:33.894 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:33.895 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:33.895 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:33.896 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:33.897 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:33.944 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:33.944 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:34.005 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:34.005 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:34.006 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.007 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.008 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:34.009 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:34.010 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.011 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.012 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.013 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.014 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932447.593883_93.1", 2021-09-06T12:47:34.014 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.014 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.014 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:27.593883Z", 2021-09-06T12:47:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.015 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:34.016 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:34.017 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:34.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.019 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:34.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:34.045 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-09-06T12:47:34.046 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-7 2021-09-06T12:47:34.063 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-8 2021-09-06T12:47:34.096 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-9 2021-09-06T12:47:34.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:34.130 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:34.131 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:34.175 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:34.175 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:34.219 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:34.235 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:34.235 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:34.235 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:34.236 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:34.236 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:34.236 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:34.236 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:34.237 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:34.237 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:34.237 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:34.237 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932454.117559_105.1", 2021-09-06T12:47:34.237 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:34.117559Z" 2021-09-06T12:47:34.238 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:34.238 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:34.238 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:34.238 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:34.239 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:34.239 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:34.240 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:34.241 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:34.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:34.242 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:34.289 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:34.290 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:34.353 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:34.354 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.355 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.356 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.357 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:34.358 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.358 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.358 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.359 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:34.359 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.359 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.359 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:34.360 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.360 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.360 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.360 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.362 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932454.054593_94.1", 2021-09-06T12:47:34.363 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.363 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.363 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.363 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:34.054593Z", 2021-09-06T12:47:34.363 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:34.364 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:34.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:34.365 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:34.365 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:34.365 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:34.366 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:34.366 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:34.366 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:34.366 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:34.366 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.367 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.367 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:34.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:34.367 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:34.368 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:34.368 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:34.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:34.368 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:34.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630932454.117559_105.1 target=1_1630932454.054593_94.1 2021-09-06T12:47:39.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.372 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:39.423 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:39.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:39.492 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:39.492 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:39.493 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:39.493 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:39.493 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:39.493 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:39.494 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.494 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:39.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.495 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:39.495 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.495 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.495 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:39.496 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.496 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:39.496 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.496 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:39.496 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.497 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.497 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:39.498 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.498 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.498 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:39.498 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.498 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.499 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.499 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:39.500 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.500 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.500 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932454.117559_105.1", 2021-09-06T12:47:39.501 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.501 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.501 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:34.117559Z", 2021-09-06T12:47:39.502 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.502 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:39.503 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.503 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.503 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:39.503 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.503 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.504 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.504 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.504 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.504 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.504 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.505 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:39.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.505 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:39.505 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:39.506 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:39.506 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.506 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:39.507 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:39.508 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:39.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.509 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:39.560 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:39.560 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:39.627 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:39.627 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:39.627 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:39.628 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:39.628 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:39.628 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:39.628 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:39.629 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:39.629 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:39.629 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:39.629 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:39.629 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:39.630 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:39.630 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:39.630 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:39.630 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:39.631 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:39.631 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:39.631 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:39.631 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.632 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:39.632 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:39.632 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:39.632 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:39.632 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:39.633 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:39.633 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:39.633 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:39.633 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:39.634 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.634 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:39.634 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.634 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:39.635 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:39.635 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:39.635 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:39.635 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:39.635 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932454.117559_105.1", 2021-09-06T12:47:39.636 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:39.636 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:39.636 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:39.636 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:34.117559Z", 2021-09-06T12:47:39.637 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:39.637 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.637 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:39.637 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:39.637 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:39.638 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:39.638 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:39.638 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:39.638 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:39.638 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:39.639 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:39.639 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.639 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:39.639 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.640 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.640 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:39.640 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:39.640 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:39.640 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:39.641 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:39.641 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:39.641 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:39.642 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:39.643 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:39.643 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.644 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:39.690 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:39.690 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:39.760 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.763 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:39.764 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.766 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:39.767 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.767 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.767 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.767 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:39.769 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.769 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.769 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932454.117559_105.1", 2021-09-06T12:47:39.769 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.770 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.770 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:34.117559Z", 2021-09-06T12:47:39.770 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:39.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:39.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:39.772 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:39.772 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:39.772 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:39.772 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:39.772 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:39.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:39.773 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:39.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.774 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:39.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:39.774 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:39.774 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:39.775 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:39.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:39.775 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:39.776 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:39.953 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:39.954 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:39.996 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:39.997 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:40.041 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:40.056 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:40.057 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:40.057 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:40.057 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:40.058 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:40.058 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:40.058 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:40.058 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:40.058 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:40.059 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:40.059 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:40.059 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:47:40.059 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:40.060 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:40.060 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:40.060 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:40.060 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:40.060 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:40.062 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:40.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:40.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:40.063 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:40.109 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:40.109 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:40.173 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:40.173 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:40.173 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:40.174 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:40.174 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:40.174 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:40.174 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:40.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:40.175 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:40.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:40.175 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:40.175 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:40.176 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:40.176 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:40.176 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:40.176 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:40.176 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:40.177 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:40.177 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:40.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:40.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:40.178 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:40.178 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:40.178 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:40.179 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:40.179 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:40.179 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:40.179 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:40.180 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:40.180 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:40.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:40.181 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:40.181 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:40.181 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:40.182 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:40.182 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.821301_107.1", 2021-09-06T12:47:40.182 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:40.183 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:40.183 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:40.183 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.821301Z", 2021-09-06T12:47:40.183 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:40.183 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:40.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:40.184 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:40.184 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:40.185 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:40.185 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:40.185 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:40.185 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:40.185 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:40.186 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:40.186 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:40.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.187 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:40.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:40.187 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:40.187 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:40.188 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:40.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:40.188 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:40.189 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1630932459.927055_112.1 target=1_1630932459.821301_107.1 2021-09-06T12:47:45.191 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.193 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:45.243 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:45.243 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:45.312 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:45.313 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.315 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:45.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.315 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:45.315 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:45.316 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.318 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.320 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.321 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.323 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.325 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:45.326 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:45.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.327 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:45.328 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:45.328 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:45.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.330 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:45.382 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:45.383 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:45.451 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:45.451 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:45.452 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:45.452 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:45.452 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:45.452 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:45.453 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:45.453 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:45.453 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:45.453 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:45.453 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:45.454 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:45.454 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:45.454 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:45.454 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:45.455 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:45.455 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:45.455 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:45.455 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:45.456 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.456 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:45.456 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:45.456 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:45.457 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:45.457 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:45.457 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:45.457 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:45.457 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:45.458 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:45.458 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.458 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:45.458 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.458 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:45.459 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:45.459 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:45.459 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:45.459 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:45.460 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:45.460 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:45.460 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:45.460 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:45.460 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:45.461 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:45.461 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.461 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:45.461 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:45.462 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:45.462 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:45.462 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:45.462 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:45.462 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:45.463 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:45.463 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:45.463 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.463 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:45.464 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.464 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.464 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:45.464 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:45.464 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:45.465 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:45.465 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:45.465 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:45.465 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:45.467 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:45.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:45.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:45.469 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:45.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:45.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:45.578 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:45.579 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:45.579 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:45.579 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:45.579 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:45.579 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:45.580 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.580 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:45.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.581 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:45.581 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.581 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.581 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:45.582 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.582 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:45.582 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.582 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:45.582 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.583 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:45.584 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.584 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.584 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:45.584 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:45.586 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.587 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:45.589 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:45.590 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:45.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:45.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:45.593 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:45.594 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:45.605 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-09-06T12:47:45.606 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-10 2021-09-06T12:47:45.625 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-11 2021-09-06T12:47:45.656 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-12 2021-09-06T12:47:45.858 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:45.858+0000 7f8ccac36700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.858 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:45.858+0000 7f8cc6c2e700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.859 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:45.858+0000 7f8cc1c24700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.859 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:45.858+0000 7f8ccc439700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.860 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:45.859+0000 7f0513c72700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.860 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:45.859+0000 7f0514473700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.860 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:45.859+0000 7f0508c5c700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.860 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:45.859+0000 7f0507c5a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.861 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:45.860+0000 7f0515475700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.861 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:45.860+0000 7f89ed781700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.861 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:45.860+0000 7f89f5791700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.862 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:45.860+0000 7f89f1789700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:45.862 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:45.860+0000 7f89ec77f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:46.263 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:46.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:46.264 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:46.330 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:46.330 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:46.454 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:46.469 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:46.469 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:46.470 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:46.470 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:46.470 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:46.470 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:46.471 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:46.471 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:46.471 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:46.471 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:46.471 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:46.472 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:47:46.472 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:46.472 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:46.472 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932465.677249_45.1", 2021-09-06T12:47:46.473 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:45.677249Z" 2021-09-06T12:47:46.473 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:46.473 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:46.474 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:46.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:46.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:46.476 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:46.565 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:46.565 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:46.635 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:46.635 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:46.635 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:46.635 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:46.636 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:46.636 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:46.636 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:46.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:46.637 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:46.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:46.637 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:46.637 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:46.638 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:46.638 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:46.638 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:46.638 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:46.639 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:46.639 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:46.639 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:46.639 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:46.640 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:46.641 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:46.641 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:46.641 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:46.641 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:46.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:46.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:46.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:46.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:46.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:46.643 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:46.643 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:46.643 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:46.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:46.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:46.644 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:46.644 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:46.644 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:46.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:46.645 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:46.645 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:46.645 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:46.646 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932465.615140_34.1", 2021-09-06T12:47:46.646 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:46.646 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:46.646 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:46.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:45.615140Z", 2021-09-06T12:47:46.647 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:46.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.647 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:46.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:46.648 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:46.648 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:46.648 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:46.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:46.649 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:46.650 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932465.677249_45.1 target=1_1630932465.615140_34.1 2021-09-06T12:47:47.277 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.277+0000 7f050f469700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.278 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.277+0000 7f0514c74700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.278 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.277+0000 7f0514473700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.278 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.277+0000 7f0507c5a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.279 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.279+0000 7f050845b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.458 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.458+0000 7f050c463700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.458 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.458+0000 7f050c463700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.458 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.458+0000 7f050c463700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.459 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:47.458+0000 7f050945d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.460 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:47.459+0000 7f8ccd43b700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.460 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:47.459+0000 7f8cce43d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.460 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:47.459+0000 7f8cc0c22700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.460 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:47.459+0000 7f8cc2c26700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:47.645 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:47.645+0000 7f8cce43d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:51.651 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.653 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:51.704 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:51.704 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:51.785 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:51.786 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:51.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:51.788 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:51.789 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:51.790 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:51.791 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:51.792 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:51.793 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:51.794 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:51.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:51.796 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932465.677249_45.1", 2021-09-06T12:47:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:51.797 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:45.677249Z", 2021-09-06T12:47:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.798 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:51.799 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:51.800 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:51.800 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:51.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:47:51.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:47:51.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.803 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:51.861 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:51.861 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:51.934 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:47:51.934 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:47:51.935 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:47:51.935 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:47:51.935 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:47:51.935 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:51.935 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:51.936 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:51.936 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:47:51.937 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:51.937 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:47:51.937 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:51.937 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:51.937 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:51.938 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:51.938 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:47:51.938 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:51.938 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:51.938 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:51.939 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.939 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:51.939 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:51.939 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:47:51.939 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:51.940 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:51.940 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:51.940 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:51.940 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:51.940 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:51.941 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:47:51.941 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:51.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.941 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:51.942 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:51.942 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:47:51.942 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:51.942 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:51.943 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:51.943 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:51.943 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:51.943 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:51.943 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:51.944 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:51.944 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.944 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:51.944 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:51.944 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932465.677249_45.1", 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:47:51.945 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:47:51.946 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:45.677249Z", 2021-09-06T12:47:51.946 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:47:51.946 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.946 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.947 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:47:51.947 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:51.947 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:47:51.947 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:47:51.947 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:47:51.948 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:51.948 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:47:51.949 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:47:51.950 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:47:51.950 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:51.951 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:51.997 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:51.998 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:52.062 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:52.062 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:52.063 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:52.064 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:52.065 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:52.066 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:52.067 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932465.677249_45.1", 2021-09-06T12:47:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:45.677249Z", 2021-09-06T12:47:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:52.075 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:52.076 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:47:52.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:52.077 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:52.077 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:52.127 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:52.128 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:52.152 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.151+0000 7f66362f8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:52.152 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.151+0000 7f66362f8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:52.152 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.151+0000 7f66362f8380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:47:52.152 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.151+0000 7f66362f8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:52.184 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.183+0000 7f66362f8380 1 bucket sync caught up with source: 2021-09-06T12:47:52.184 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:52.184 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:52.185 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:52.183+0000 7f66362f8380 0 bucket checkpoint complete 2021-09-06T12:47:52.200 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-10 zones={test-zone2, test-zone1} 2021-09-06T12:47:52.207 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:52.211 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:52.212 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:52.212 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:52.212 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:52.213 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:52.259 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:52.259 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:52.260 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:52.260 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:52.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:52.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:52.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:52.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:52.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:52.715 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:52.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:52.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:52.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:53.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:53.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:53.219 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:53.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-10 zones={test-zone2, test-zone1} 2021-09-06T12:47:53.319 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:53.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:53.320 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:53.377 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:53.378 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:53.403 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.403+0000 7fbea744a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:53.403 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.403+0000 7fbea744a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:53.404 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.403+0000 7fbea744a380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:47:53.404 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.403+0000 7fbea744a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:53.436 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.437+0000 7fbea744a380 1 bucket sync caught up with source: 2021-09-06T12:47:53.437 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:53.437 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:53.437 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:53.437+0000 7fbea744a380 0 bucket checkpoint complete 2021-09-06T12:47:53.453 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-10 zones={test-zone3, test-zone1} 2021-09-06T12:47:53.460 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:53.464 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:53.464 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:53.465 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:53.465 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:53.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:53.470 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:53.470 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:53.471 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:53.471 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:53.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:53.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:53.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:53.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:53.776 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:53.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:53.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:54.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:54.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:54.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:54.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:54.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:54.478 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-10 zones={test-zone3, test-zone1} 2021-09-06T12:47:54.479 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:54.479 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-10', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:54.480 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-10 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:54.537 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:54.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:54.565 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.565+0000 7fcd3e752380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.565+0000 7fcd3e752380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.565+0000 7fcd3e752380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:47:54.566 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.565+0000 7fcd3e752380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:54.599 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.599+0000 7fcd3e752380 1 bucket sync caught up with source: 2021-09-06T12:47:54.600 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:54.600 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.9.5, , 00000000001.3.5, , , , 00000000001.5.5, 00000000001.5.5, , ] 2021-09-06T12:47:54.600 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:54.599+0000 7fcd3e752380 0 bucket checkpoint complete 2021-09-06T12:47:54.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:54.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:54.617 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:54.673 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:54.673 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:54.698 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.699+0000 7fb3b4f0c380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:54.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.699+0000 7fb3b4f0c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:54.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.699+0000 7fb3b4f0c380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:47:54.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.699+0000 7fb3b4f0c380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:54.741 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.741+0000 7fb3b4f0c380 1 bucket sync caught up with source: 2021-09-06T12:47:54.741 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.94.5, , 00000000001.24.5, , , , 00000000001.113.5, 00000000001.43.5, , ] 2021-09-06T12:47:54.742 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.94.5, , 00000000001.24.5, , , , 00000000001.113.5, 00000000001.43.5, , ] 2021-09-06T12:47:54.742 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:54.741+0000 7fb3b4f0c380 0 bucket checkpoint complete 2021-09-06T12:47:54.758 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-11 zones={test-zone1, test-zone2} 2021-09-06T12:47:54.764 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:54.769 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:54.770 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:54.770 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:54.771 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:54.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:54.775 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:54.776 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:54.776 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:54.777 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:54.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:54.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:54.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:55.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:55.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:55.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:55.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:55.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:55.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:55.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:55.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:55.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:55.791 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-11 zones={test-zone1, test-zone2} 2021-09-06T12:47:55.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:55.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-11', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:55.793 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-11 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:55.854 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:55.855 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:55.881 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.882+0000 7f9ddfada380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:55.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.882+0000 7f9ddfada380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:55.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.882+0000 7f9ddfada380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:47:55.882 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.882+0000 7f9ddfada380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:55.916 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.916+0000 7f9ddfada380 1 bucket sync caught up with source: 2021-09-06T12:47:55.916 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.94.5, , 00000000001.24.5, , , , 00000000001.113.5, 00000000001.43.5, , ] 2021-09-06T12:47:55.916 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.94.5, , 00000000001.24.5, , , , 00000000001.113.5, 00000000001.43.5, , ] 2021-09-06T12:47:55.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:55.916+0000 7f9ddfada380 0 bucket checkpoint complete 2021-09-06T12:47:55.932 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-11 zones={test-zone3, test-zone2} 2021-09-06T12:47:55.938 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:55.943 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:55.943 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:55.944 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:55.944 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:55.945 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:55.950 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:55.951 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:55.951 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:55.951 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:55.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:56.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:56.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:56.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:56.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:56.358 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:56.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:56.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:56.607 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:56.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:56.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:56.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:56.950 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-11 zones={test-zone3, test-zone2} 2021-09-06T12:47:56.951 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:56.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:56.952 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:57.011 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:57.011 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:57.037 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.038+0000 7f315c455380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:57.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.038+0000 7f315c455380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:57.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.038+0000 7f315c455380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:47:57.038 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.038+0000 7f315c455380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:57.075 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.076+0000 7f315c455380 1 bucket sync caught up with source: 2021-09-06T12:47:57.075 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.25.5, , 00000000001.10.5, , , , 00000000001.44.5, 00000000001.33.5, , ] 2021-09-06T12:47:57.075 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.25.5, , 00000000001.10.5, , , , 00000000001.44.5, 00000000001.33.5, , ] 2021-09-06T12:47:57.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:47:57.076+0000 7f315c455380 0 bucket checkpoint complete 2021-09-06T12:47:57.091 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-12 zones={test-zone1, test-zone3} 2021-09-06T12:47:57.098 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:57.101 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:57.102 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:57.102 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:57.103 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:57.103 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:57.107 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:57.107 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:57.108 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:57.108 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:57.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:57.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:57.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:57.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:57.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:57.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:57.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:57.658 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:57.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:57.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:57.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:58.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:58.106 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-12 zones={test-zone1, test-zone3} 2021-09-06T12:47:58.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:58.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-12', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:47:58.108 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-12 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:47:58.166 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:58.167 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:58.194 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.194+0000 7f0410ee4380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:47:58.195 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.194+0000 7f0410ee4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:47:58.195 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.194+0000 7f0410ee4380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:47:58.195 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.194+0000 7f0410ee4380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:47:58.229 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.229+0000 7f0410ee4380 1 bucket sync caught up with source: 2021-09-06T12:47:58.230 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.25.5, , 00000000001.10.5, , , , 00000000001.44.5, 00000000001.33.5, , ] 2021-09-06T12:47:58.230 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.25.5, , 00000000001.10.5, , , , 00000000001.44.5, 00000000001.33.5, , ] 2021-09-06T12:47:58.230 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:47:58.229+0000 7f0410ee4380 0 bucket checkpoint complete 2021-09-06T12:47:58.246 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-12 zones={test-zone2, test-zone3} 2021-09-06T12:47:58.253 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:47:58.257 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:58.257 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:58.258 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:58.259 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:58.259 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:47:58.263 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-09-06T12:47:58.264 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-09-06T12:47:58.264 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-09-06T12:47:58.264 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:47:58.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:58.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:58.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-09-06T12:47:58.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:58.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:58.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-09-06T12:47:58.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:58.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:58.974 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-09-06T12:47:59.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:59.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:59.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:47:59.323 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-12 zones={test-zone2, test-zone3} 2021-09-06T12:47:59.323 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-09-06T12:47:59.324 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-13 2021-09-06T12:47:59.345 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-14 2021-09-06T12:47:59.381 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-15 2021-09-06T12:47:59.462 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:59.463+0000 7f89f378d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:59.464 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T12:47:59.462+0000 7f8cc1423700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:59.465 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:59.462+0000 7f0516c78700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:59.488 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:47:59.489+0000 7f89f4f90700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:59.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:59.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:59.560 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:59.608 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:59.608 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:59.656 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:47:59.672 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:47:59.672 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:59.672 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:59.672 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:47:59.673 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:59.673 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:59.673 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:47:59.673 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:47:59.673 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:59.674 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:59.674 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:59.674 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:47:59.674 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:47:59.674 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:47:59.675 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932479.402599_60.1", 2021-09-06T12:47:59.675 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:59.402599Z" 2021-09-06T12:47:59.675 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:47:59.675 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:47:59.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:47:59.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:59.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:47:59.678 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:47:59.683 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:47:59.682+0000 7f051146d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T12:47:59.730 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:47:59.731 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:47:59.813 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:47:59.813 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:47:59.813 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:47:59.814 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:47:59.814 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:47:59.814 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:47:59.814 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:59.815 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:59.815 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:47:59.815 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:59.815 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:47:59.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:59.816 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:59.816 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:47:59.816 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:59.816 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:47:59.817 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:59.817 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:47:59.817 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:59.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:59.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:59.818 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:47:59.818 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:59.818 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:59.819 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:47:59.819 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:59.819 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:59.819 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:59.819 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:47:59.820 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:59.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:59.820 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:59.821 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:47:59.821 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:59.821 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:59.821 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:47:59.821 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:59.822 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:59.822 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:59.822 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:47:59.822 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:59.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:59.823 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:47:59.823 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:47:59.823 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:47:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:47:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932465.677249_45.1", 2021-09-06T12:47:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:47:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:47:59.824 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:47:59.825 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:45.677249Z", 2021-09-06T12:47:59.825 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:47:59.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.825 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:47:59.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:47:59.826 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:47:59.826 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:47:59.826 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:47:59.827 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:47:59.827 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:47:59.828 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932479.402599_60.1 target=1_1630932465.677249_45.1 2021-09-06T12:48:04.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:04.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:04.830 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:04.903 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:04.904 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:04.983 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:48:04.983 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:48:04.984 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:48:04.984 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:48:04.984 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:48:04.984 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:04.985 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:04.985 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:04.985 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:48:04.985 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:04.986 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:48:04.986 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:04.986 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:04.986 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:04.987 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:04.987 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:48:04.987 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:04.987 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:04.988 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:04.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.988 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:04.988 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:04.988 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:48:04.989 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:04.989 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:04.989 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:48:04.989 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:04.990 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:04.990 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:04.990 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:48:04.990 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:04.991 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:04.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:04.991 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:48:04.991 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:04.992 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:04.992 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:04.992 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:04.992 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:04.993 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:04.993 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:04.993 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:04.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:04.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:04.994 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:48:04.994 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:04.994 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:04.995 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932479.402599_60.1", 2021-09-06T12:48:04.995 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:04.995 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:04.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:04.996 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:59.402599Z", 2021-09-06T12:48:04.996 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:04.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.997 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:48:04.997 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:04.997 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:48:04.997 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:48:04.998 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:48:04.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:04.998 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:48:04.999 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:48:05.000 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:48:05.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:05.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:05.002 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:05.046 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:05.047 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:05.107 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:48:05.108 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:48:05.108 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:48:05.108 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:48:05.109 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:48:05.109 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:05.109 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:05.109 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:05.109 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:48:05.110 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:05.110 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:48:05.110 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:05.110 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:05.111 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:05.111 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:05.111 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:48:05.111 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:05.112 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:05.112 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:05.112 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.112 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:05.113 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:05.113 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:48:05.113 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:05.113 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:05.114 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:48:05.114 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:05.114 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:05.114 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:05.115 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:48:05.115 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:05.115 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.115 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:05.115 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:05.116 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:48:05.116 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:05.116 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:05.116 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:05.117 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:05.117 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:05.117 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:05.117 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:05.117 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:05.118 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.118 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:05.118 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:05.118 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:48:05.119 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:05.119 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:05.119 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932479.333878_49.1", 2021-09-06T12:48:05.119 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:05.119 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:05.120 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:05.120 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:59.333878Z", 2021-09-06T12:48:05.120 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:05.120 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.121 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.121 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:48:05.121 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:05.121 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:48:05.121 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:48:05.122 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:48:05.122 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:05.122 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:48:05.123 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932479.402599_60.1 target=1_1630932479.333878_49.1 2021-09-06T12:48:10.124 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:10.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:10.126 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:10.176 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:10.176 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:10.248 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:48:10.248 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:48:10.248 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:48:10.249 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:48:10.249 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:48:10.249 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:10.249 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:10.250 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:10.250 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:48:10.250 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:10.250 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:48:10.250 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:10.251 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:10.251 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:10.251 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:10.251 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:48:10.251 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:10.252 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:10.252 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:10.252 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.252 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:10.253 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:10.253 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:48:10.253 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:10.253 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:10.253 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:48:10.254 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:10.254 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:10.254 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:10.254 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:48:10.255 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:10.255 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.255 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:10.255 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:10.255 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:48:10.256 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:10.256 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:10.256 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:10.256 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:10.256 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:10.257 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:10.257 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:10.257 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:10.257 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.258 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:10.258 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:10.258 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:48:10.258 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:10.258 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:10.259 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932479.402599_60.1", 2021-09-06T12:48:10.259 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:10.259 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:10.259 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:10.259 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:59.402599Z", 2021-09-06T12:48:10.260 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:10.260 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.260 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.260 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:48:10.261 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:10.261 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:48:10.261 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:48:10.261 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:48:10.261 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:10.262 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:48:10.263 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:48:10.264 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:48:10.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:10.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:10.266 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:10.316 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:10.316 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:10.385 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:48:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:48:10.386 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:48:10.386 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:48:10.386 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:48:10.386 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:10.387 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:10.387 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:10.387 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:48:10.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:10.387 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:48:10.388 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:10.388 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:10.388 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:10.388 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:10.388 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:48:10.389 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:10.389 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:10.389 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:10.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.390 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:10.390 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:10.390 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:48:10.390 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:10.391 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:10.392 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:48:10.392 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:10.392 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:10.393 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:10.393 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:48:10.393 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:10.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:10.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:10.394 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:48:10.394 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:10.395 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:10.395 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:10.395 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:10.395 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:10.395 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:10.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:10.396 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:10.396 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:10.396 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:10.397 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:48:10.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:10.397 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:10.397 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932479.402599_60.1", 2021-09-06T12:48:10.398 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:10.398 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:10.398 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:10.398 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:59.402599Z", 2021-09-06T12:48:10.398 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:10.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.399 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:48:10.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:48:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:48:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:48:10.400 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:10.400 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:48:10.402 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:48:10.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:10.403 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:10.404 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:10.456 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:10.457 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:10.479 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.478+0000 7f50e3b01380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:10.479 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.478+0000 7f50e3b01380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:10.480 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.478+0000 7f50e3b01380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:10.480 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.478+0000 7f50e3b01380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:10.512 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.511+0000 7f50e3b01380 1 bucket sync caught up with source: 2021-09-06T12:48:10.512 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:10.512 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:10.513 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:10.511+0000 7f50e3b01380 0 bucket checkpoint complete 2021-09-06T12:48:10.527 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-13 zones={test-zone2, test-zone1} 2021-09-06T12:48:10.533 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:10.579 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:10.580 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:10.583 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:10.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:10.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:10.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:10.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-13 zones={test-zone2, test-zone1} 2021-09-06T12:48:10.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:10.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:10.885 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:10.940 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:10.941 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:10.965 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.966+0000 7f5ff0cc8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:10.966 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.966+0000 7f5ff0cc8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:10.966 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.966+0000 7f5ff0cc8380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:10.966 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.966+0000 7f5ff0cc8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:10.999 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.999+0000 7f5ff0cc8380 1 bucket sync caught up with source: 2021-09-06T12:48:10.999 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:11.000 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:11.000 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:10.999+0000 7f5ff0cc8380 0 bucket checkpoint complete 2021-09-06T12:48:11.015 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-13 zones={test-zone3, test-zone1} 2021-09-06T12:48:11.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:11.067 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:11.068 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:11.071 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:11.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:11.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:11.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:11.411 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-13 zones={test-zone3, test-zone1} 2021-09-06T12:48:11.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:11.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:11.412 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:11.471 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:11.471 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:11.497 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.497+0000 7f4d7458a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:11.498 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.497+0000 7f4d7458a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:11.498 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.497+0000 7f4d7458a380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:48:11.499 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.497+0000 7f4d7458a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:11.532 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.531+0000 7f4d7458a380 1 bucket sync caught up with source: 2021-09-06T12:48:11.532 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:11.532 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:11.533 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:11.531+0000 7f4d7458a380 0 bucket checkpoint complete 2021-09-06T12:48:11.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:11.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:11.549 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:11.602 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:11.603 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:11.627 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.628+0000 7f30c4fc1380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:11.628 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.628+0000 7f30c4fc1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:11.628 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.628+0000 7f30c4fc1380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:11.628 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.628+0000 7f30c4fc1380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:11.661 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.661+0000 7f30c4fc1380 1 bucket sync caught up with source: 2021-09-06T12:48:11.661 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-06T12:48:11.661 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.55.5, , , , , , , , , ] 2021-09-06T12:48:11.661 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:11.661+0000 7f30c4fc1380 0 bucket checkpoint complete 2021-09-06T12:48:11.678 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-14 zones={test-zone1, test-zone2} 2021-09-06T12:48:11.685 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:11.731 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:11.732 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:11.735 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:11.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:11.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:11.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-14 zones={test-zone1, test-zone2} 2021-09-06T12:48:12.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:12.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:12.032 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:12.083 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:12.084 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:12.107 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.107+0000 7f1f2f123380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:12.107 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.107+0000 7f1f2f123380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:12.108 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.107+0000 7f1f2f123380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:12.108 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.107+0000 7f1f2f123380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:12.139 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.139+0000 7f1f2f123380 1 bucket sync caught up with source: 2021-09-06T12:48:12.140 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-06T12:48:12.140 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.55.5, , , , , , , , , ] 2021-09-06T12:48:12.140 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.139+0000 7f1f2f123380 0 bucket checkpoint complete 2021-09-06T12:48:12.154 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-14 zones={test-zone3, test-zone2} 2021-09-06T12:48:12.160 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:12.207 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:12.208 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:12.212 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:12.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.515 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-14 zones={test-zone3, test-zone2} 2021-09-06T12:48:12.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:12.516 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:12.517 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:12.569 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:12.569 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:12.592 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.593+0000 7f52526ff380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:12.593 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.593+0000 7f52526ff380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:12.593 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.593+0000 7f52526ff380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:12.593 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.593+0000 7f52526ff380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:12.633 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.634+0000 7f52526ff380 1 bucket sync caught up with source: 2021-09-06T12:48:12.634 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.74.5, , , , , , , , , ] 2021-09-06T12:48:12.634 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000001.74.5, , , , , , , , , ] 2021-09-06T12:48:12.634 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:12.634+0000 7f52526ff380 0 bucket checkpoint complete 2021-09-06T12:48:12.651 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-15 zones={test-zone1, test-zone3} 2021-09-06T12:48:12.657 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:12.704 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:12.705 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:12.708 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:12.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:12.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:13.042 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-15 zones={test-zone1, test-zone3} 2021-09-06T12:48:13.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:13.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:13.044 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:13.103 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:13.103 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:13.129 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.129+0000 7f5cdc473380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:13.129 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.129+0000 7f5cdc473380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.129+0000 7f5cdc473380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:13.130 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.129+0000 7f5cdc473380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:13.165 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.164+0000 7f5cdc473380 1 bucket sync caught up with source: 2021-09-06T12:48:13.165 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.74.5, , , , , , , , , ] 2021-09-06T12:48:13.165 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000001.74.5, , , , , , , , , ] 2021-09-06T12:48:13.165 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.164+0000 7f5cdc473380 0 bucket checkpoint complete 2021-09-06T12:48:13.181 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-15 zones={test-zone2, test-zone3} 2021-09-06T12:48:13.187 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:13.235 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:13.236 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:13.239 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-09-06T12:48:13.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:13.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:13.438 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-09-06T12:48:13.534 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-15 zones={test-zone2, test-zone3} 2021-09-06T12:48:13.545 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:13.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:13.547 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:13.600 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:13.601 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.624+0000 7f2b7ed4c380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.624+0000 7f2b7ed4c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.624+0000 7f2b7ed4c380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:13.625 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.624+0000 7f2b7ed4c380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:13.658 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:13.658+0000 7f2b7ed4c380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:13.658 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000001.13.5, , , , , , , , , ] 2021-09-06T12:48:13.659 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:18.660 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:18.659+0000 7f2b7ed4c380 1 bucket sync caught up with source: 2021-09-06T12:48:18.660 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:18.661 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:18.661 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:18.659+0000 7f2b7ed4c380 0 bucket checkpoint complete 2021-09-06T12:48:18.680 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-13 zones={test-zone2, test-zone1} 2021-09-06T12:48:18.688 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:18.735 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:18.786 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-13 zones={test-zone2, test-zone1} 2021-09-06T12:48:18.787 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:18.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:18.788 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:18.849 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:18.849 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:18.877 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.876+0000 7f843e757380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:18.877 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.876+0000 7f843e757380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:18.878 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.876+0000 7f843e757380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:18.878 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.876+0000 7f843e757380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:18.911 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.911+0000 7f843e757380 1 bucket sync caught up with source: 2021-09-06T12:48:18.912 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:18.912 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:18.912 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:18.911+0000 7f843e757380 0 bucket checkpoint complete 2021-09-06T12:48:18.927 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-13 zones={test-zone3, test-zone1} 2021-09-06T12:48:18.933 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:18.979 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:19.030 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-13 zones={test-zone3, test-zone1} 2021-09-06T12:48:19.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:19.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-13', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:19.032 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-13 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:19.089 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:19.090 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:19.116 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.115+0000 7f34db52b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:19.116 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.115+0000 7f34db52b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:19.116 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.115+0000 7f34db52b380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:48:19.117 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.115+0000 7f34db52b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:19.149 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.149+0000 7f34db52b380 1 bucket sync caught up with source: 2021-09-06T12:48:19.150 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:19.150 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000002.15.5, , , , , , , , , ] 2021-09-06T12:48:19.150 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:19.149+0000 7f34db52b380 0 bucket checkpoint complete 2021-09-06T12:48:19.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:19.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:19.176 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:19.227 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:19.228 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:19.250 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:19.250+0000 7f9b8e7b5380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:19.251 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:19.250+0000 7f9b8e7b5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:19.251 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:19.250+0000 7f9b8e7b5380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:19.251 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:19.250+0000 7f9b8e7b5380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:19.285 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:19.285+0000 7f9b8e7b5380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:19.285 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.55.5, , , , , , , , , ] 2021-09-06T12:48:19.286 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.137.5, , , , , , , , , ] 2021-09-06T12:48:24.287 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.287+0000 7f9b8e7b5380 1 bucket sync caught up with source: 2021-09-06T12:48:24.288 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000002.137.5, , , , , , , , , ] 2021-09-06T12:48:24.289 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.137.5, , , , , , , , , ] 2021-09-06T12:48:24.289 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.287+0000 7f9b8e7b5380 0 bucket checkpoint complete 2021-09-06T12:48:24.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-14 zones={test-zone1, test-zone2} 2021-09-06T12:48:24.320 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:24.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:24.426 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-14 zones={test-zone1, test-zone2} 2021-09-06T12:48:24.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:24.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-14', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:24.428 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-14 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:24.485 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:24.486 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:24.511 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.511+0000 7f59b131b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:24.512 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.511+0000 7f59b131b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:24.512 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.511+0000 7f59b131b380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:24.512 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.511+0000 7f59b131b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:24.565 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.565+0000 7f59b131b380 1 bucket sync caught up with source: 2021-09-06T12:48:24.566 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000002.137.5, , , , , , , , , ] 2021-09-06T12:48:24.566 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.137.5, , , , , , , , , ] 2021-09-06T12:48:24.566 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.565+0000 7f59b131b380 0 bucket checkpoint complete 2021-09-06T12:48:24.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-14 zones={test-zone3, test-zone2} 2021-09-06T12:48:24.588 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:24.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:24.687 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-14 zones={test-zone3, test-zone2} 2021-09-06T12:48:24.697 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:24.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:24.698 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:24.750 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:24.750 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:24.773 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.773+0000 7ff5fcd7a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:24.774 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.773+0000 7ff5fcd7a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:24.774 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.773+0000 7ff5fcd7a380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:24.774 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.773+0000 7ff5fcd7a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:24.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:24.808+0000 7ff5fcd7a380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:24.809 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000001.74.5, , , , , , , , , ] 2021-09-06T12:48:24.809 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-06T12:48:29.810 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:29.809+0000 7ff5fcd7a380 1 bucket sync caught up with source: 2021-09-06T12:48:29.811 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-06T12:48:29.811 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-06T12:48:29.811 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:29.809+0000 7ff5fcd7a380 0 bucket checkpoint complete 2021-09-06T12:48:29.830 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-15 zones={test-zone1, test-zone3} 2021-09-06T12:48:29.837 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:29.887 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:29.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-15 zones={test-zone1, test-zone3} 2021-09-06T12:48:29.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:29.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-15', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:29.940 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-15 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:29.998 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:29.998 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:30.023 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.023+0000 7f46c73bb380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:30.024 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.023+0000 7f46c73bb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:30.024 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.023+0000 7f46c73bb380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:30.024 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.023+0000 7f46c73bb380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:30.058 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.058+0000 7f46c73bb380 1 bucket sync caught up with source: 2021-09-06T12:48:30.058 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000002.186.5, , , , , , , , , ] 2021-09-06T12:48:30.059 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000002.186.5, , , , , , , , , ] 2021-09-06T12:48:30.059 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:30.058+0000 7f46c73bb380 0 bucket checkpoint complete 2021-09-06T12:48:30.075 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-15 zones={test-zone2, test-zone3} 2021-09-06T12:48:30.081 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:30.127 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:30.179 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-15 zones={test-zone2, test-zone3} 2021-09-06T12:48:30.180 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-09-06T12:48:30.181 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-16 2021-09-06T12:48:30.205 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-17 2021-09-06T12:48:30.240 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-18 2021-09-06T12:48:30.313 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:30.313 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:30.314 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:30.358 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:30.358 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:30.413 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:48:30.430 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:48:30.431 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:30.431 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:30.431 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:48:30.431 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:30.432 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:30.432 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:48:30.432 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:48:30.432 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:30.432 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:30.433 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:30.433 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:48:30.433 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:30.433 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:30.434 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932510.309812_82.1", 2021-09-06T12:48:30.434 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:48:30.309812Z" 2021-09-06T12:48:30.434 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:30.434 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:48:30.435 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:48:30.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:30.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:30.437 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:30.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:30.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:30.548 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:48:30.548 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:48:30.548 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:48:30.549 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:48:30.549 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:48:30.549 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:30.549 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:30.550 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:30.550 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:48:30.550 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:30.550 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:48:30.551 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:30.551 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:30.551 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:30.551 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:30.551 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:48:30.552 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:30.552 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:30.552 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:30.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:30.553 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:30.553 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:48:30.553 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:30.553 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:30.554 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:48:30.554 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:30.554 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:30.555 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:30.555 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:48:30.555 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:30.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:30.556 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:30.556 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:48:30.556 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:30.556 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:30.557 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:30.557 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:30.557 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:30.557 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:30.558 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:30.558 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:30.558 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:30.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:30.559 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:48:30.559 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:30.559 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:30.559 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932510.192885_65.1", 2021-09-06T12:48:30.560 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:30.560 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:30.560 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:30.560 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:48:30.192885Z", 2021-09-06T12:48:30.561 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:30.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.561 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.561 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:48:30.561 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:30.562 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:48:30.562 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:48:30.562 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:48:30.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:30.563 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:48:30.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932510.309812_82.1 target=1_1630932510.192885_65.1 2021-09-06T12:48:35.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.566 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:35.617 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:35.617 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:35.686 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:48:35.686 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:48:35.686 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:48:35.686 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:48:35.687 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:48:35.687 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:35.687 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.688 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:48:35.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.688 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:48:35.688 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.689 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.689 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:35.689 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.689 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:48:35.689 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:35.690 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:48:35.691 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.692 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.692 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:48:35.692 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.693 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:48:35.693 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.693 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.694 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:35.694 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.694 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.694 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:35.695 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.695 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.695 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.695 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.696 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:48:35.696 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.696 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.696 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932510.309812_82.1", 2021-09-06T12:48:35.696 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.697 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.697 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:48:30.309812Z", 2021-09-06T12:48:35.697 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.698 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:48:35.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.698 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:48:35.699 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:48:35.699 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:48:35.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.699 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:48:35.700 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:48:35.701 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:48:35.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.703 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:35.753 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:35.753 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:35.821 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:48:35.821 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:48:35.821 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:48:35.821 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:48:35.822 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:48:35.822 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:35.822 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:35.822 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:35.823 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:48:35.823 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:35.823 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:48:35.823 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:35.823 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:35.824 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:35.824 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:35.824 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:48:35.824 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:35.824 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:35.825 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:35.825 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.825 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:35.825 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:35.826 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:48:35.826 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:35.826 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:35.826 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:48:35.826 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:35.827 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:35.827 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:35.827 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:48:35.827 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:35.828 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.828 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:35.828 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:35.828 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:48:35.828 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:35.829 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:35.829 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:35.829 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:35.829 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:35.830 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:35.830 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:35.830 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:35.830 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.830 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:35.831 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:48:35.831 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:48:35.831 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:48:35.831 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:48:35.832 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932510.309812_82.1", 2021-09-06T12:48:35.832 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:48:35.832 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:48:35.832 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:48:35.833 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:48:30.309812Z", 2021-09-06T12:48:35.833 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:48:35.833 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.833 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.833 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:48:35.834 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:48:35.834 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:48:35.834 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:48:35.834 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:48:35.835 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:48:35.835 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:48:35.836 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:48:35.836 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:48:35.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:48:35.838 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:48:35.883 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:35.883 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:35.956 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:48:35.957 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:48:35.957 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:48:35.957 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:48:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:48:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:48:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.958 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:48:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:48:35.959 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.960 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.960 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:48:35.960 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.960 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:48:35.960 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.961 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:48:35.961 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.961 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:48:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.962 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:48:35.963 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.963 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.963 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.963 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:48:35.964 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.964 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.964 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:48:35.965 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.965 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.965 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:48:35.965 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.965 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.966 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.966 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:48:35.966 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.966 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:48:35.967 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:48:35.967 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:48:35.967 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:48:35.968 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932510.309812_82.1", 2021-09-06T12:48:35.968 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:48:35.968 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:48:35.968 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:48:35.969 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:48:30.309812Z", 2021-09-06T12:48:35.969 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:48:35.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.969 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:48:35.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:48:35.970 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:48:35.970 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:48:35.970 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:48:35.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:48:35.971 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:48:35.972 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:48:36.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:36.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:36.020 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:36.070 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:36.070 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:36.096 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:36.096+0000 7ff2e2553380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:36.096 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:36.096+0000 7ff2e2553380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:36.096 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:36.096+0000 7ff2e2553380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:36.096 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:36.096+0000 7ff2e2553380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:36.129 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:36.129+0000 7ff2e2553380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:36.129 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , , , ] 2021-09-06T12:48:36.129 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-09-06T12:48:41.131 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.131+0000 7ff2e2553380 1 bucket sync caught up with source: 2021-09-06T12:48:41.131 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:48:41.132 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000002.9.10, , , , , ] 2021-09-06T12:48:41.132 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.131+0000 7ff2e2553380 0 bucket checkpoint complete 2021-09-06T12:48:41.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.152 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:41.210 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:41.210 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:41.237 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.236+0000 7f8a52d54380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:41.237 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.236+0000 7f8a52d54380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:41.237 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.236+0000 7f8a52d54380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:41.237 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.236+0000 7f8a52d54380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:41.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.272+0000 7f8a52d54380 1 bucket sync caught up with source: 2021-09-06T12:48:41.272 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:48:41.272 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:48:41.273 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.272+0000 7f8a52d54380 0 bucket checkpoint complete 2021-09-06T12:48:41.288 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.289 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.289 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:41.346 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:41.346 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:41.373 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.373+0000 7fb72ede3380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:41.373 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.373+0000 7fb72ede3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:41.373 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.373+0000 7fb72ede3380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:48:41.373 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.373+0000 7fb72ede3380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:41.407 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.408+0000 7fb72ede3380 1 bucket sync caught up with source: 2021-09-06T12:48:41.407 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:48:41.408 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:48:41.408 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:41.408+0000 7fb72ede3380 0 bucket checkpoint complete 2021-09-06T12:48:41.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:41.472 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:41.525 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:41.526 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:41.550 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.549+0000 7f2abae41380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:41.550 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.549+0000 7f2abae41380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:41.550 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.549+0000 7f2abae41380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:41.550 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.549+0000 7f2abae41380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:41.600 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:41.599+0000 7f2abae41380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:41.600 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , , , ] 2021-09-06T12:48:41.600 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000002.155.10, , , , , ] 2021-09-06T12:48:46.602 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.602+0000 7f2abae41380 1 bucket sync caught up with source: 2021-09-06T12:48:46.602 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.163.9, , , , , ] 2021-09-06T12:48:46.602 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000002.155.10, , , , , ] 2021-09-06T12:48:46.603 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.602+0000 7f2abae41380 0 bucket checkpoint complete 2021-09-06T12:48:46.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:46.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:46.623 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:46.679 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:46.680 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:46.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.707+0000 7f72e7c3a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:46.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.707+0000 7f72e7c3a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:46.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.707+0000 7f72e7c3a380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:46.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.707+0000 7f72e7c3a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:46.754 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.754+0000 7f72e7c3a380 1 bucket sync caught up with source: 2021-09-06T12:48:46.754 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.163.9, , , , , ] 2021-09-06T12:48:46.754 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000006.163.9, , , , , ] 2021-09-06T12:48:46.755 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.754+0000 7f72e7c3a380 0 bucket checkpoint complete 2021-09-06T12:48:46.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:46.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:46.824 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:46.876 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:46.877 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:46.900 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.901+0000 7fc150054380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:46.901 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.901+0000 7fc150054380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:46.901 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.901+0000 7fc150054380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:46.901 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.901+0000 7fc150054380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:46.952 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:46.952+0000 7fc150054380 1 waiting for incremental sync to catch up: 2021-09-06T12:48:46.952 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , , , ] 2021-09-06T12:48:46.952 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000002.288.10, , , , , ] 2021-09-06T12:48:51.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:51.954+0000 7fc150054380 1 bucket sync caught up with source: 2021-09-06T12:48:51.954 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.296.9, , , , , ] 2021-09-06T12:48:51.954 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000002.288.10, , , , , ] 2021-09-06T12:48:51.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:51.954+0000 7fc150054380 0 bucket checkpoint complete 2021-09-06T12:48:51.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:51.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:51.976 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:52.030 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:52.031 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:52.055 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.056+0000 7fa657c5b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:52.056 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.056+0000 7fa657c5b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:52.056 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.056+0000 7fa657c5b380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:52.056 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.056+0000 7fa657c5b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:52.090 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.090+0000 7fa657c5b380 1 bucket sync caught up with source: 2021-09-06T12:48:52.090 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.296.9, , , , , ] 2021-09-06T12:48:52.090 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.296.9, , , , , ] 2021-09-06T12:48:52.090 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:52.090+0000 7fa657c5b380 0 bucket checkpoint complete 2021-09-06T12:48:53.632 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.634 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:53.686 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:53.686 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:53.724 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.724+0000 7f396b273380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:53.724 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.724+0000 7f396b273380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:53.724 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.724+0000 7f396b273380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:53.725 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.724+0000 7f396b273380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:53.762 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.762+0000 7f396b273380 1 bucket sync caught up with source: 2021-09-06T12:48:53.762 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:53.762 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:53.762 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:53.762+0000 7f396b273380 0 bucket checkpoint complete 2021-09-06T12:48:53.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.778 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.779 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:53.832 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:53.833 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:53.861 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.862+0000 7fd879ac5380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:53.862 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.862+0000 7fd879ac5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:53.862 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.862+0000 7fd879ac5380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:53.862 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.862+0000 7fd879ac5380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:53.905 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.906+0000 7fd879ac5380 1 bucket sync caught up with source: 2021-09-06T12:48:53.906 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:53.906 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:53.906 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:53.906+0000 7fd879ac5380 0 bucket checkpoint complete 2021-09-06T12:48:53.926 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:53.928 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:53.980 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:53.980 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:54.004 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.004+0000 7fd532d43380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:54.004 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.004+0000 7fd532d43380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:54.004 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.004+0000 7fd532d43380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:48:54.004 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.004+0000 7fd532d43380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:54.036 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.036+0000 7fd532d43380 1 bucket sync caught up with source: 2021-09-06T12:48:54.036 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:54.036 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.25.5, , 00000000005.16.5, , 00000000009.23.9, , , , , ] 2021-09-06T12:48:54.036 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.036+0000 7fd532d43380 0 bucket checkpoint complete 2021-09-06T12:48:54.051 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.052 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:54.103 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:54.104 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:54.127 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.127+0000 7f6e2b95f380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:54.127 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.127+0000 7f6e2b95f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:54.127 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.127+0000 7f6e2b95f380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:54.128 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.127+0000 7f6e2b95f380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:54.179 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.179+0000 7f6e2b95f380 1 bucket sync caught up with source: 2021-09-06T12:48:54.179 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.225.9, , 00000000010.151.5, , 00000000010.190.5, , , , , ] 2021-09-06T12:48:54.180 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.225.9, , 00000000010.151.5, , 00000000010.190.5, , , , , ] 2021-09-06T12:48:54.180 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.179+0000 7f6e2b95f380 0 bucket checkpoint complete 2021-09-06T12:48:54.197 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.197 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.198 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:54.248 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:54.249 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:54.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.272+0000 7f0904e9d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:54.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.272+0000 7f0904e9d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:54.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.272+0000 7f0904e9d380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:54.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.272+0000 7f0904e9d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:54.306 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.307+0000 7f0904e9d380 1 bucket sync caught up with source: 2021-09-06T12:48:54.307 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.225.9, , 00000000010.151.5, , 00000000010.190.5, , , , , ] 2021-09-06T12:48:54.307 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.225.9, , 00000000010.151.5, , 00000000010.190.5, , , , , ] 2021-09-06T12:48:54.307 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.307+0000 7f0904e9d380 0 bucket checkpoint complete 2021-09-06T12:48:54.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.323 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:54.373 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:54.374 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:54.397 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.397+0000 7f332c6ba380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:54.397 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.397+0000 7f332c6ba380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:54.397 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.397+0000 7f332c6ba380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:54.398 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.397+0000 7f332c6ba380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:54.429 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.429+0000 7f332c6ba380 1 bucket sync caught up with source: 2021-09-06T12:48:54.429 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.138.9, , 00000000005.113.5, , 00000000014.200.5, , , , , ] 2021-09-06T12:48:54.429 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.138.9, , 00000000005.113.5, , 00000000014.200.5, , , , , ] 2021-09-06T12:48:54.429 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:54.429+0000 7f332c6ba380 0 bucket checkpoint complete 2021-09-06T12:48:54.444 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.445 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:54.446 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:54.497 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:54.498 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:54.522 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.522+0000 7f8ff4ff1380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:54.522 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.522+0000 7f8ff4ff1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:54.522 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.522+0000 7f8ff4ff1380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:54.522 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.522+0000 7f8ff4ff1380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:54.553 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.554+0000 7f8ff4ff1380 1 bucket sync caught up with source: 2021-09-06T12:48:54.553 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000003.138.9, , 00000000005.113.5, , 00000000014.200.5, , , , , ] 2021-09-06T12:48:54.554 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000003.138.9, , 00000000005.113.5, , 00000000014.200.5, , , , , ] 2021-09-06T12:48:54.554 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:54.554+0000 7f8ff4ff1380 0 bucket checkpoint complete 2021-09-06T12:48:54.569 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone2, test-zone1} 2021-09-06T12:48:54.576 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:54.579 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:54.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:54.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:54.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:54.581 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:54.584 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:54.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:54.585 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:54.586 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:54.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:54.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:54.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:54.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:54.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:54.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:54.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:55.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:55.109 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:55.206 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:55.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:55.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:55.454 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone2, test-zone1} 2021-09-06T12:48:55.455 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone3, test-zone1} 2021-09-06T12:48:55.461 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:55.507 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:55.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:55.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:55.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:55.509 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:55.513 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:55.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:55.514 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:55.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:55.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:55.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:55.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:55.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:55.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:55.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:55.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:55.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:56.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:56.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:56.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:56.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:56.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone3, test-zone1} 2021-09-06T12:48:56.387 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone3, test-zone2} 2021-09-06T12:48:56.393 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:56.397 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:56.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:56.397 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:56.398 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:56.398 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:56.402 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:56.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:56.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:56.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:56.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:56.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:56.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:56.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:56.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:56.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:56.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:56.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:56.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:57.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:57.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:57.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:57.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone3, test-zone2} 2021-09-06T12:48:57.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.288 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:57.345 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:57.346 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:57.372 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.372+0000 7ffb8717a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:57.372 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.372+0000 7ffb8717a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:57.372 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.372+0000 7ffb8717a380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:57.373 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.372+0000 7ffb8717a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:57.406 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.407+0000 7ffb8717a380 1 bucket sync caught up with source: 2021-09-06T12:48:57.406 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.407 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.407 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.407+0000 7ffb8717a380 0 bucket checkpoint complete 2021-09-06T12:48:57.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.424 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:57.483 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:57.483 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:57.509 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.510+0000 7fdf08c96380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:57.510 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.510+0000 7fdf08c96380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:57.510 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.510+0000 7fdf08c96380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:57.510 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.510+0000 7fdf08c96380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:57.549 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.549+0000 7fdf08c96380 1 bucket sync caught up with source: 2021-09-06T12:48:57.549 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.550 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.550 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.549+0000 7fdf08c96380 0 bucket checkpoint complete 2021-09-06T12:48:57.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.568 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:57.623 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:57.624 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:57.651 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.651+0000 7f52279a6380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:57.651 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.651+0000 7f52279a6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:57.651 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.651+0000 7f52279a6380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:48:57.651 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.651+0000 7f52279a6380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:57.685 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.686+0000 7f52279a6380 1 bucket sync caught up with source: 2021-09-06T12:48:57.686 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.686 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.27.5, , 00000000003.9.9, , 00000000007.19.9, , , , , ] 2021-09-06T12:48:57.686 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:57.686+0000 7f52279a6380 0 bucket checkpoint complete 2021-09-06T12:48:57.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.702 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:57.754 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:57.755 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:57.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.780+0000 7f072797b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:57.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.780+0000 7f072797b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:57.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.780+0000 7f072797b380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:57.780 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.780+0000 7f072797b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:57.814 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.815+0000 7f072797b380 1 bucket sync caught up with source: 2021-09-06T12:48:57.815 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000006.245.5, , 00000000010.114.5, , 00000000009.169.9, , , , , ] 2021-09-06T12:48:57.815 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000006.245.5, , 00000000010.114.5, , 00000000009.169.9, , , , , ] 2021-09-06T12:48:57.815 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.815+0000 7f072797b380 0 bucket checkpoint complete 2021-09-06T12:48:57.839 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.841 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:57.891 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:57.892 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:57.914 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.915+0000 7f123b1e8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:57.914 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.915+0000 7f123b1e8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:57.915 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.915+0000 7f123b1e8380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:48:57.915 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.915+0000 7f123b1e8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:57.954 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.955+0000 7f123b1e8380 1 bucket sync caught up with source: 2021-09-06T12:48:57.954 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000006.245.5, , 00000000010.114.5, , 00000000009.169.9, , , , , ] 2021-09-06T12:48:57.955 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000006.245.5, , 00000000010.114.5, , 00000000009.169.9, , , , , ] 2021-09-06T12:48:57.955 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:57.955+0000 7f123b1e8380 0 bucket checkpoint complete 2021-09-06T12:48:57.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:57.973 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:58.023 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:58.023 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:58.046 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.047+0000 7f263ee75380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:58.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.047+0000 7f263ee75380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:58.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.047+0000 7f263ee75380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:48:58.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.047+0000 7f263ee75380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:58.095 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.095+0000 7f263ee75380 1 bucket sync caught up with source: 2021-09-06T12:48:58.095 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.139.9, , 00000000003.105.9, , 00000000014.178.5, , , , , ] 2021-09-06T12:48:58.096 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.139.9, , 00000000003.105.9, , 00000000014.178.5, , , , , ] 2021-09-06T12:48:58.096 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:48:58.095+0000 7f263ee75380 0 bucket checkpoint complete 2021-09-06T12:48:58.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:58.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:48:58.113 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:48:58.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:48:58.172 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:48:58.198 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.198+0000 7f75c9aab380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:48:58.198 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.198+0000 7f75c9aab380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:48:58.198 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.198+0000 7f75c9aab380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:48:58.198 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.198+0000 7f75c9aab380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:48:58.232 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.233+0000 7f75c9aab380 1 bucket sync caught up with source: 2021-09-06T12:48:58.232 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000003.139.9, , 00000000003.105.9, , 00000000014.178.5, , , , , ] 2021-09-06T12:48:58.232 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000003.139.9, , 00000000003.105.9, , 00000000014.178.5, , , , , ] 2021-09-06T12:48:58.233 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:48:58.233+0000 7f75c9aab380 0 bucket checkpoint complete 2021-09-06T12:48:58.249 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone2, test-zone1} 2021-09-06T12:48:58.255 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:58.258 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:58.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:58.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:58.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:58.260 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:58.263 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:58.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:58.264 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:58.265 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:58.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:58.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:58.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:58.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:58.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:58.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:58.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:58.689 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:58.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:58.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:58.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:59.072 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:59.170 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone2, test-zone1} 2021-09-06T12:48:59.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone3, test-zone1} 2021-09-06T12:48:59.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:48:59.223 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:59.223 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:59.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:59.224 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:59.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:48:59.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:48:59.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:48:59.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:48:59.230 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:48:59.237 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:59.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:59.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:48:59.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:59.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:59.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:48:59.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:59.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:59.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:48:59.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:48:59.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:00.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:00.102 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone3, test-zone1} 2021-09-06T12:49:00.103 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone3, test-zone2} 2021-09-06T12:49:00.110 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:00.114 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:00.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:00.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:00.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:00.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:00.163 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:00.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:00.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:00.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:00.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:00.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:00.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:00.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:00.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:00.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:00.591 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:00.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:00.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:00.842 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:00.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:00.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:01.090 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone3, test-zone2} 2021-09-06T12:49:01.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.091 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.148 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.148 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.175 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.176+0000 7fc7d8f9b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.176 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.176+0000 7fc7d8f9b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.176 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.176+0000 7fc7d8f9b380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:01.176 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.176+0000 7fc7d8f9b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.210 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.211+0000 7fc7d8f9b380 1 bucket sync caught up with source: 2021-09-06T12:49:01.210 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.211 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.211 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.211+0000 7fc7d8f9b380 0 bucket checkpoint complete 2021-09-06T12:49:01.227 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.228 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.285 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.285 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.312 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.313+0000 7f1def556380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.312 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.313+0000 7f1def556380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.312 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.313+0000 7f1def556380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:01.313 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.313+0000 7f1def556380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.346 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.347+0000 7f1def556380 1 bucket sync caught up with source: 2021-09-06T12:49:01.347 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.347 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.347 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.347+0000 7f1def556380 0 bucket checkpoint complete 2021-09-06T12:49:01.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.364 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.416 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.416 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.442+0000 7f5600c6d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.441 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.442+0000 7f5600c6d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.442+0000 7f5600c6d380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:01.442 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.442+0000 7f5600c6d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.473 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.474+0000 7f5600c6d380 1 bucket sync caught up with source: 2021-09-06T12:49:01.474 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.474 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000010.27.5, , 00000000003.26.9, , 00000000007.31.9, , , , , ] 2021-09-06T12:49:01.474 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.474+0000 7f5600c6d380 0 bucket checkpoint complete 2021-09-06T12:49:01.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.490 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.541 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.541 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.564 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.565+0000 7f2831d2e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.564 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.565+0000 7f2831d2e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.565 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.565+0000 7f2831d2e380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:01.565 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.565+0000 7f2831d2e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.614 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.615+0000 7f2831d2e380 1 bucket sync caught up with source: 2021-09-06T12:49:01.615 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.188.9, , 00000000010.350.5, , 00000000010.365.5, , , , , ] 2021-09-06T12:49:01.615 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.188.9, , 00000000010.350.5, , 00000000010.365.5, , , , , ] 2021-09-06T12:49:01.615 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.615+0000 7f2831d2e380 0 bucket checkpoint complete 2021-09-06T12:49:01.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.633 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.682 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.683 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.706+0000 7fedf2c4b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.706+0000 7fedf2c4b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.706+0000 7fedf2c4b380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:01.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.706+0000 7fedf2c4b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.744 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.744+0000 7fedf2c4b380 1 bucket sync caught up with source: 2021-09-06T12:49:01.744 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.188.9, , 00000000010.350.5, , 00000000010.365.5, , , , , ] 2021-09-06T12:49:01.744 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.188.9, , 00000000010.350.5, , 00000000010.365.5, , , , , ] 2021-09-06T12:49:01.744 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.744+0000 7fedf2c4b380 0 bucket checkpoint complete 2021-09-06T12:49:01.759 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.761 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.815 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.815 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.841 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.841+0000 7f6e2c056380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.841 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.841+0000 7f6e2c056380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.842 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.841+0000 7f6e2c056380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:01.842 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.841+0000 7f6e2c056380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:01.877 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.877+0000 7f6e2c056380 1 bucket sync caught up with source: 2021-09-06T12:49:01.877 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000003.220.9, , 00000000006.324.5, , 00000000016.329.5, , , , , ] 2021-09-06T12:49:01.877 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000003.220.9, , 00000000006.324.5, , 00000000016.329.5, , , , , ] 2021-09-06T12:49:01.877 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:01.877+0000 7f6e2c056380 0 bucket checkpoint complete 2021-09-06T12:49:01.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:01.896 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:01.954 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:01.954 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:01.980 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.980+0000 7f3262314380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:01.980 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.980+0000 7f3262314380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:01.980 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.980+0000 7f3262314380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:01.981 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:01.980+0000 7f3262314380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:02.013 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:02.013+0000 7f3262314380 1 bucket sync caught up with source: 2021-09-06T12:49:02.013 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000003.220.9, , 00000000006.324.5, , 00000000016.329.5, , , , , ] 2021-09-06T12:49:02.013 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000003.220.9, , 00000000006.324.5, , 00000000016.329.5, , , , , ] 2021-09-06T12:49:02.014 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:02.013+0000 7f3262314380 0 bucket checkpoint complete 2021-09-06T12:49:02.029 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone2, test-zone1} 2021-09-06T12:49:02.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:02.040 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:02.040 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:02.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:02.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:02.041 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:02.045 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:02.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:02.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:02.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:02.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:02.061 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:02.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:02.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:02.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:02.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:02.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:02.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:02.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:02.662 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:02.713 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:02.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:02.914 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone2, test-zone1} 2021-09-06T12:49:02.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone3, test-zone1} 2021-09-06T12:49:02.920 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:02.967 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:02.967 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:02.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:02.968 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:02.969 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:02.973 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:02.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:02.973 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:02.974 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:02.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:02.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:03.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:03.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:03.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:03.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:03.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:03.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:03.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:03.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:03.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:03.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:03.846 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone3, test-zone1} 2021-09-06T12:49:03.847 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone3, test-zone2} 2021-09-06T12:49:03.853 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:03.857 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:03.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:03.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:03.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:03.859 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:03.863 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:03.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:03.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:03.864 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:03.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:03.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:03.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:03.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:04.036 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:04.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:04.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:04.285 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:04.390 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:04.486 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:04.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:04.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:04.738 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone3, test-zone2} 2021-09-06T12:49:05.245 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:05.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:05.246 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:05.304 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:05.304 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:05.334 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.335+0000 7f72f7a4e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:05.334 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.335+0000 7f72f7a4e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:05.334 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.335+0000 7f72f7a4e380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:05.335 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.335+0000 7f72f7a4e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:05.371 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.371+0000 7f72f7a4e380 1 bucket sync caught up with source: 2021-09-06T12:49:05.371 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:05.372 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:05.372 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:05.371+0000 7f72f7a4e380 0 bucket checkpoint complete 2021-09-06T12:49:05.388 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:05.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:05.390 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:05.441 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:05.442 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:05.465 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:05.465+0000 7ff1af709380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:05.465 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:05.465+0000 7ff1af709380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:05.465 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:05.465+0000 7ff1af709380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:05.466 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:05.465+0000 7ff1af709380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:06.505 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.505+0000 7ff1af709380 1 bucket sync caught up with source: 2021-09-06T12:49:06.505 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:06.506 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:06.506 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.505+0000 7ff1af709380 0 bucket checkpoint complete 2021-09-06T12:49:06.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.526 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:06.584 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:06.584 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:06.610 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.610+0000 7f815b035380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:06.611 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.610+0000 7f815b035380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:06.611 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.610+0000 7f815b035380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:06.611 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.610+0000 7f815b035380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:06.645 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.644+0000 7f815b035380 1 bucket sync caught up with source: 2021-09-06T12:49:06.645 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:06.645 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.32.5, , 00000000006.20.5, , 00000000010.26.5, , , , , ] 2021-09-06T12:49:06.645 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:06.644+0000 7f815b035380 0 bucket checkpoint complete 2021-09-06T12:49:06.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.662 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:06.720 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:06.721 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:06.747 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.748+0000 7f0be297c380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:06.748 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.748+0000 7f0be297c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:06.748 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.748+0000 7f0be297c380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:06.748 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.748+0000 7f0be297c380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:06.789 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.790+0000 7f0be297c380 1 bucket sync caught up with source: 2021-09-06T12:49:06.790 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.228.5, , 00000000011.156.5, , 00000000011.194.5, , , , , ] 2021-09-06T12:49:06.790 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.228.5, , 00000000011.156.5, , 00000000011.194.5, , , , , ] 2021-09-06T12:49:06.790 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.790+0000 7f0be297c380 0 bucket checkpoint complete 2021-09-06T12:49:06.806 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.807 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:06.861 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:06.861 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:06.884 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.884+0000 7f9c14eda380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:06.885 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.884+0000 7f9c14eda380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:06.885 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.884+0000 7f9c14eda380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:06.885 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.884+0000 7f9c14eda380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:06.927 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.928+0000 7f9c14eda380 1 bucket sync caught up with source: 2021-09-06T12:49:06.927 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.228.5, , 00000000011.156.5, , 00000000011.194.5, , , , , ] 2021-09-06T12:49:06.928 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.228.5, , 00000000011.156.5, , 00000000011.194.5, , , , , ] 2021-09-06T12:49:06.928 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:06.928+0000 7f9c14eda380 0 bucket checkpoint complete 2021-09-06T12:49:06.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:06.945 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:06.995 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:06.995 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:07.018 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.018+0000 7f203a1d3380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:07.018 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.018+0000 7f203a1d3380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:07.018 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.018+0000 7f203a1d3380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:07.019 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.018+0000 7f203a1d3380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:07.050 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.050+0000 7f203a1d3380 1 bucket sync caught up with source: 2021-09-06T12:49:07.050 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.141.5, , 00000000006.117.5, , 00000000015.203.5, , , , , ] 2021-09-06T12:49:07.051 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.141.5, , 00000000006.117.5, , 00000000015.203.5, , , , , ] 2021-09-06T12:49:07.051 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:07.050+0000 7f203a1d3380 0 bucket checkpoint complete 2021-09-06T12:49:07.066 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:07.067 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-16', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:07.068 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-16 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:07.124 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:07.124 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:07.149 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.149+0000 7f1b37ad0380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:07.150 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.149+0000 7f1b37ad0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:07.150 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.149+0000 7f1b37ad0380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:07.150 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.149+0000 7f1b37ad0380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:07.182 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.182+0000 7f1b37ad0380 1 bucket sync caught up with source: 2021-09-06T12:49:07.183 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000004.141.5, , 00000000006.117.5, , 00000000015.203.5, , , , , ] 2021-09-06T12:49:07.183 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000004.141.5, , 00000000006.117.5, , 00000000015.203.5, , , , , ] 2021-09-06T12:49:07.183 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:07.182+0000 7f1b37ad0380 0 bucket checkpoint complete 2021-09-06T12:49:07.198 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone2, test-zone1} 2021-09-06T12:49:07.205 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:07.208 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:07.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:07.209 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:07.210 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:07.210 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:07.214 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:07.214 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:07.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:07.215 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:07.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:07.228 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:07.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:07.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:07.388 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:07.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:07.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:07.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:07.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:07.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:07.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:07.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:08.082 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone2, test-zone1} 2021-09-06T12:49:08.083 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone3, test-zone1} 2021-09-06T12:49:08.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:08.092 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:08.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:08.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:08.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:08.093 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:08.097 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:08.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:08.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:08.098 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:08.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:08.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:08.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:08.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:08.273 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:08.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:08.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:08.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:08.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:08.718 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:08.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:08.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:08.966 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone3, test-zone1} 2021-09-06T12:49:08.966 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-16 zones={test-zone3, test-zone2} 2021-09-06T12:49:08.972 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:08.976 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:08.976 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:08.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:08.977 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:08.978 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:09.023 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:09.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:09.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:09.024 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:09.075 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:09.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:09.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:09.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:09.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:09.345 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:09.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:09.493 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:09.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:09.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:09.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:09.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:09.946 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-16 zones={test-zone3, test-zone2} 2021-09-06T12:49:09.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:09.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:09.948 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.004 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.005 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.030 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.029+0000 7f7074c14380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.030 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.029+0000 7f7074c14380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.031 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.029+0000 7f7074c14380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:10.031 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.029+0000 7f7074c14380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.065 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.065+0000 7f7074c14380 1 bucket sync caught up with source: 2021-09-06T12:49:10.066 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.066 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.066 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.065+0000 7f7074c14380 0 bucket checkpoint complete 2021-09-06T12:49:10.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.083 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.142 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.142 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.170 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.171+0000 7f6d780da380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.170 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.171+0000 7f6d780da380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.171+0000 7f6d780da380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:10.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.171+0000 7f6d780da380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.205 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.205+0000 7f6d780da380 1 bucket sync caught up with source: 2021-09-06T12:49:10.205 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.205 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.206 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.205+0000 7f6d780da380 0 bucket checkpoint complete 2021-09-06T12:49:10.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.223 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.275 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.275 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.298 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.298+0000 7f27a89db380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.299 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.298+0000 7f27a89db380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.299 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.298+0000 7f27a89db380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:10.299 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.298+0000 7f27a89db380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.330 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.330+0000 7f27a89db380 1 bucket sync caught up with source: 2021-09-06T12:49:10.331 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.331 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.33.5, , 00000000004.12.5, , 00000000008.22.5, , , , , ] 2021-09-06T12:49:10.331 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.330+0000 7f27a89db380 0 bucket checkpoint complete 2021-09-06T12:49:10.346 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.347 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.399 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.400 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.423 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.423+0000 7fb2f4e10380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.423 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.423+0000 7fb2f4e10380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.423 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.423+0000 7fb2f4e10380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:10.424 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.423+0000 7fb2f4e10380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.458 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.458+0000 7fb2f4e10380 1 bucket sync caught up with source: 2021-09-06T12:49:10.458 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000007.249.5, , 00000000011.121.5, , 00000000010.172.5, , , , , ] 2021-09-06T12:49:10.458 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000007.249.5, , 00000000011.121.5, , 00000000010.172.5, , , , , ] 2021-09-06T12:49:10.459 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.458+0000 7fb2f4e10380 0 bucket checkpoint complete 2021-09-06T12:49:10.477 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.479 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.529 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.529 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.552 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.553+0000 7f290ba33380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.553 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.553+0000 7f290ba33380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.553 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.553+0000 7f290ba33380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:10.553 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.553+0000 7f290ba33380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.589 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.590+0000 7f290ba33380 1 bucket sync caught up with source: 2021-09-06T12:49:10.589 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000007.249.5, , 00000000011.121.5, , 00000000010.172.5, , , , , ] 2021-09-06T12:49:10.590 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000007.249.5, , 00000000011.121.5, , 00000000010.172.5, , , , , ] 2021-09-06T12:49:10.590 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.590+0000 7f290ba33380 0 bucket checkpoint complete 2021-09-06T12:49:10.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.606 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.656 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.656 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.680 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.680+0000 7f0177a53380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.680 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.680+0000 7f0177a53380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.680 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.680+0000 7f0177a53380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:10.681 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.680+0000 7f0177a53380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.722 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.723+0000 7f0177a53380 1 bucket sync caught up with source: 2021-09-06T12:49:10.723 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.142.5, , 00000000004.108.5, , 00000000015.181.5, , , , , ] 2021-09-06T12:49:10.723 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.142.5, , 00000000004.108.5, , 00000000015.181.5, , , , , ] 2021-09-06T12:49:10.723 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:10.723+0000 7f0177a53380 0 bucket checkpoint complete 2021-09-06T12:49:10.741 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-17', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:10.743 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-17 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:10.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:10.796 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:10.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.820+0000 7f03e8723380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:10.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.820+0000 7f03e8723380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:10.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.820+0000 7f03e8723380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:10.821 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.820+0000 7f03e8723380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:10.854 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.854+0000 7f03e8723380 1 bucket sync caught up with source: 2021-09-06T12:49:10.854 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000004.142.5, , 00000000004.108.5, , 00000000015.181.5, , , , , ] 2021-09-06T12:49:10.855 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000004.142.5, , 00000000004.108.5, , 00000000015.181.5, , , , , ] 2021-09-06T12:49:10.855 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:10.854+0000 7f03e8723380 0 bucket checkpoint complete 2021-09-06T12:49:10.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone2, test-zone1} 2021-09-06T12:49:10.876 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:10.880 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:10.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:10.881 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:10.882 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:10.882 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:10.886 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:10.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:10.887 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:10.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:10.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:10.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:11.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:11.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:11.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:11.257 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:11.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:11.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:11.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:11.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:11.659 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:11.758 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone2, test-zone1} 2021-09-06T12:49:11.759 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone3, test-zone1} 2021-09-06T12:49:11.765 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:11.768 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:11.769 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:11.770 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:11.815 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:11.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:11.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:11.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:11.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:11.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:11.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:11.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:12.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:12.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:12.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:12.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:12.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:12.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:12.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:12.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:12.719 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone3, test-zone1} 2021-09-06T12:49:12.720 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-17 zones={test-zone3, test-zone2} 2021-09-06T12:49:12.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:12.731 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:12.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:12.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:12.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:12.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:12.736 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:12.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:12.737 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:12.738 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:12.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:12.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:12.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:12.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:12.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:13.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:13.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:13.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:13.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:13.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:13.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:13.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:13.620 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-17 zones={test-zone3, test-zone2} 2021-09-06T12:49:13.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.624 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:13.683 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:13.684 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:13.709 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.708+0000 7fccf0031380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:13.709 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.708+0000 7fccf0031380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:13.710 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.708+0000 7fccf0031380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:13.710 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.708+0000 7fccf0031380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:13.750 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.749+0000 7fccf0031380 1 bucket sync caught up with source: 2021-09-06T12:49:13.751 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:13.752 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:13.752 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:13.749+0000 7fccf0031380 0 bucket checkpoint complete 2021-09-06T12:49:13.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.773 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.774 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:13.839 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:13.840 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:13.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.867+0000 7f071bbb1380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:13.867 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.867+0000 7f071bbb1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:13.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.867+0000 7f071bbb1380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:13.868 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.867+0000 7f071bbb1380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:13.906 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.906+0000 7f071bbb1380 1 bucket sync caught up with source: 2021-09-06T12:49:13.907 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:13.907 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:13.908 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:13.906+0000 7f071bbb1380 0 bucket checkpoint complete 2021-09-06T12:49:13.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:13.926 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:13.982 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:13.983 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:14.005 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:14.004+0000 7ffac1fc8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:14.006 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:14.004+0000 7ffac1fc8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:14.006 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:14.004+0000 7ffac1fc8380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:14.007 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:14.004+0000 7ffac1fc8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:15.039 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.038+0000 7ffac1fc8380 1 bucket sync caught up with source: 2021-09-06T12:49:15.040 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:15.040 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000011.34.5, , 00000000004.36.5, , 00000000008.34.5, , , , , ] 2021-09-06T12:49:15.040 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.038+0000 7ffac1fc8380 0 bucket checkpoint complete 2021-09-06T12:49:15.059 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.061 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:15.120 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:15.120 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:15.147 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.148+0000 7f8472ccb380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:15.147 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.148+0000 7f8472ccb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:15.148 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.148+0000 7f8472ccb380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:15.148 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.148+0000 7f8472ccb380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:15.187 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.187+0000 7f8472ccb380 1 bucket sync caught up with source: 2021-09-06T12:49:15.187 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.191.5, , 00000000011.370.5, , 00000000011.372.5, , , , , ] 2021-09-06T12:49:15.188 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.191.5, , 00000000011.370.5, , 00000000011.372.5, , , , , ] 2021-09-06T12:49:15.188 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.187+0000 7f8472ccb380 0 bucket checkpoint complete 2021-09-06T12:49:15.203 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.206 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:15.258 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:15.259 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:15.282 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.282+0000 7f0feff60380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:15.282 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.282+0000 7f0feff60380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:15.282 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.282+0000 7f0feff60380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:15.283 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.282+0000 7f0feff60380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:15.316 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.317+0000 7f0feff60380 1 bucket sync caught up with source: 2021-09-06T12:49:15.317 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000004.191.5, , 00000000011.370.5, , 00000000011.372.5, , , , , ] 2021-09-06T12:49:15.317 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000004.191.5, , 00000000011.370.5, , 00000000011.372.5, , , , , ] 2021-09-06T12:49:15.317 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.317+0000 7f0feff60380 0 bucket checkpoint complete 2021-09-06T12:49:15.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.336 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:15.386 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:15.386 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:15.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.409+0000 7f9629f5b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:15.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.409+0000 7f9629f5b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:15.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.409+0000 7f9629f5b380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:15.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.409+0000 7f9629f5b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:15.460 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.459+0000 7f9629f5b380 1 bucket sync caught up with source: 2021-09-06T12:49:15.460 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, 00000000005.227.5, , 00000000007.334.5, , 00000000017.332.5, , , , , ] 2021-09-06T12:49:15.460 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, 00000000005.227.5, , 00000000007.334.5, , 00000000017.332.5, , , , , ] 2021-09-06T12:49:15.460 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:15.459+0000 7f9629f5b380 0 bucket checkpoint complete 2021-09-06T12:49:15.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-18', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:15.481 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-18 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:15.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:15.539 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:15.567 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.566+0000 7f36382e7380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:15.567 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.566+0000 7f36382e7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:15.567 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.566+0000 7f36382e7380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:15.568 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.566+0000 7f36382e7380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:15.601 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.601+0000 7f36382e7380 1 bucket sync caught up with source: 2021-09-06T12:49:15.602 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, 00000000005.227.5, , 00000000007.334.5, , 00000000017.332.5, , , , , ] 2021-09-06T12:49:15.602 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, 00000000005.227.5, , 00000000007.334.5, , 00000000017.332.5, , , , , ] 2021-09-06T12:49:15.602 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:15.601+0000 7f36382e7380 0 bucket checkpoint complete 2021-09-06T12:49:15.619 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone2, test-zone1} 2021-09-06T12:49:15.625 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:15.628 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:15.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:15.629 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:15.630 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:15.630 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:15.634 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:15.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:15.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:15.635 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:15.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:15.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:15.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:15.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:15.808 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:15.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:16.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:16.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:16.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:16.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:16.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:16.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:16.494 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone2, test-zone1} 2021-09-06T12:49:16.494 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone3, test-zone1} 2021-09-06T12:49:16.501 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:16.505 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:16.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:16.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:16.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:16.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:16.510 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:16.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:16.511 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:16.512 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:16.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:16.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:16.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:16.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:16.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:16.793 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:16.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:16.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:17.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:17.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:17.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:17.296 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:17.390 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone3, test-zone1} 2021-09-06T12:49:17.391 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-18 zones={test-zone3, test-zone2} 2021-09-06T12:49:17.397 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:17.401 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:17.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:17.402 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:17.403 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:17.403 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:17.451 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:17.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-09-06T12:49:17.452 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-09-06T12:49:17.453 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-09-06T12:49:17.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:17.512 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:17.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:17.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:17.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:17.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-09-06T12:49:17.878 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:17.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:18.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-09-06T12:49:18.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:18.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:18.289 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-09-06T12:49:18.386 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-18 zones={test-zone3, test-zone2} 2021-09-06T12:49:18.387 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-09-06T12:49:18.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:18.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:18.425 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:49:18.472 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:18.472 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:18.538 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:49:18.558 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:49:18.558 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:18.559 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:49:18.559 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:49:18.559 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:18.559 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:18.560 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:49:18.560 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:49:18.560 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:18.560 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:18.560 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:49:18.561 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:49:18.561 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:18.561 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:18.561 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932558.424027_88.1", 2021-09-06T12:49:18.561 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:18.424027Z" 2021-09-06T12:49:18.562 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:18.562 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:49:18.563 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:49:18.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:18.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:18.565 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:49:18.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:18.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:18.686 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:49:18.686 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:49:18.687 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:49:18.687 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:49:18.687 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:49:18.687 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:49:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:49:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:18.688 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:49:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:49:18.689 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:18.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:49:18.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:18.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:49:18.690 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:18.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:18.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:18.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:49:18.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:18.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:18.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:49:18.692 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:18.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:18.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:18.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:49:18.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:18.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:18.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:18.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:49:18.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:18.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:18.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:49:18.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:18.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:18.695 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:18.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:49:18.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:18.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:18.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:18.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:49:18.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:18.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:18.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932558.401711_83.1", 2021-09-06T12:49:18.698 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:18.698 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:18.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:18.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:18.401711Z", 2021-09-06T12:49:18.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:18.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.699 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:49:18.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:18.700 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:49:18.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:49:18.700 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:49:18.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:18.700 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:49:18.702 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932558.424027_88.1 target=1_1630932558.401711_83.1 2021-09-06T12:49:23.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.703 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.704 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:49:23.757 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:23.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:23.827 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:49:23.827 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:49:23.828 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:49:23.828 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:49:23.828 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:49:23.828 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:49:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:49:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:23.829 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:49:23.830 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:23.830 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:23.830 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:49:23.830 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:49:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:49:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:23.831 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:23.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:23.832 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:49:23.832 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:23.832 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:49:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:23.833 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:49:23.834 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:23.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:23.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:23.835 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:49:23.835 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:23.835 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:23.835 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:49:23.835 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:23.836 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:23.836 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:23.836 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:49:23.836 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:23.836 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.837 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:23.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:23.837 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:49:23.837 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:23.837 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:23.838 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932558.424027_88.1", 2021-09-06T12:49:23.838 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:23.838 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:23.838 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:23.838 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:18.424027Z", 2021-09-06T12:49:23.839 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:23.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.839 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:23.840 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:49:23.841 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:49:23.842 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:49:23.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.843 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:49:23.892 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:23.892 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:23.961 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:49:23.961 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:49:23.961 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:49:23.962 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:49:23.962 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:49:23.962 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:49:23.962 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:49:23.962 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:23.963 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:49:23.963 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:23.963 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:49:23.963 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:49:23.964 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:49:23.964 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:49:23.964 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:49:23.964 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:49:23.964 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:49:23.965 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:49:23.965 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:49:23.965 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.965 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:23.966 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:23.966 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:49:23.966 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:49:23.966 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:49:23.966 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:49:23.967 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:49:23.967 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:49:23.967 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:49:23.967 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:49:23.968 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:49:23.968 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.968 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:23.968 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:23.968 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:49:23.969 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:49:23.969 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:49:23.969 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:49:23.969 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:49:23.969 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:49:23.970 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:49:23.970 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:49:23.970 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:49:23.970 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.970 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:23.971 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:49:23.971 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:49:23.971 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:49:23.971 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:49:23.971 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932558.424027_88.1", 2021-09-06T12:49:23.972 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:49:23.972 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:49:23.972 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:49:23.972 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:18.424027Z", 2021-09-06T12:49:23.972 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:49:23.973 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.973 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.973 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:49:23.973 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:49:23.974 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:49:23.974 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:49:23.974 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:49:23.974 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:49:23.974 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:49:23.975 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:49:23.976 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:49:23.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.977 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:49:23.977 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:49:24.021 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:24.021 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:24.083 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:49:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:49:24.083 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:49:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:49:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:49:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:49:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:24.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:49:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:49:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:24.085 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:49:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:49:24.086 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:49:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:24.087 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:49:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:49:24.088 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:49:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:24.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:49:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:24.090 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:49:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:24.091 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:49:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:49:24.092 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:49:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:49:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:49:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932558.424027_88.1", 2021-09-06T12:49:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:49:24.093 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:49:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:49:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:18.424027Z", 2021-09-06T12:49:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:49:24.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:49:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:49:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:49:24.095 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:49:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:49:24.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:49:24.096 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:49:24.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:49:24.143 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:24.143 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:24.144 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:24.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:24.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:24.219 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:24.218+0000 7f099b09d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:24.219 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:24.218+0000 7f099b09d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:24.219 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:24.218+0000 7f099b09d380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:24.219 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:24.218+0000 7f099b09d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:24.251 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:24.251+0000 7f099b09d380 1 waiting for incremental sync to catch up: 2021-09-06T12:49:24.252 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , , , ] 2021-09-06T12:49:24.252 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2021-09-06T12:49:29.230 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:49:29.229+0000 7f04fc029700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:49:29.240 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:49:29.240+0000 7f89dff66700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:49:29.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.254+0000 7f099b09d380 1 bucket sync caught up with source: 2021-09-06T12:49:29.254 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:49:29.255 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000002.10.10, , , , , ] 2021-09-06T12:49:29.255 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.254+0000 7f099b09d380 0 bucket checkpoint complete 2021-09-06T12:49:29.275 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.277 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:29.336 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:29.336 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:29.363 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.363+0000 7f47cd389380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:29.363 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.363+0000 7f47cd389380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:29.364 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.363+0000 7f47cd389380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:29.364 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.363+0000 7f47cd389380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:29.401 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.401+0000 7f47cd389380 1 bucket sync caught up with source: 2021-09-06T12:49:29.402 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:49:29.402 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:49:29.402 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.401+0000 7f47cd389380 0 bucket checkpoint complete 2021-09-06T12:49:29.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.422 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:29.479 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:29.479 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:29.505 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.505+0000 7f1feaffd380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.505+0000 7f1feaffd380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.505+0000 7f1feaffd380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:29.506 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.505+0000 7f1feaffd380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:29.539 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.539+0000 7f1feaffd380 1 bucket sync caught up with source: 2021-09-06T12:49:29.540 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:49:29.540 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.17.9, , , , , ] 2021-09-06T12:49:29.540 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:29.539+0000 7f1feaffd380 0 bucket checkpoint complete 2021-09-06T12:49:29.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.557 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:29.608 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:29.608 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:29.631 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.631+0000 7f14d291b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:29.632 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.631+0000 7f14d291b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:29.632 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.631+0000 7f14d291b380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:29.632 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.631+0000 7f14d291b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:29.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.673+0000 7f14d291b380 1 bucket sync caught up with source: 2021-09-06T12:49:29.674 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.337.9, , , , , ] 2021-09-06T12:49:29.674 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.337.9, , , , , ] 2021-09-06T12:49:29.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.673+0000 7f14d291b380 0 bucket checkpoint complete 2021-09-06T12:49:29.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.690 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:29.740 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:29.740 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:29.765 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.764+0000 7f28eface380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:29.765 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.764+0000 7f28eface380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:29.765 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.764+0000 7f28eface380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:29.765 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.764+0000 7f28eface380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:29.798 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.797+0000 7f28eface380 1 bucket sync caught up with source: 2021-09-06T12:49:29.798 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.337.9, , , , , ] 2021-09-06T12:49:29.798 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.337.9, , , , , ] 2021-09-06T12:49:29.799 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.797+0000 7f28eface380 0 bucket checkpoint complete 2021-09-06T12:49:29.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.816 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:29.872 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:29.872 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:29.895 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.895+0000 7f6ee9418380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:29.895 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.895+0000 7f6ee9418380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:29.896 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.895+0000 7f6ee9418380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:29.896 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.895+0000 7f6ee9418380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:29.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.938+0000 7f6ee9418380 1 bucket sync caught up with source: 2021-09-06T12:49:29.939 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.141.9, , , , , ] 2021-09-06T12:49:29.939 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.141.9, , , , , ] 2021-09-06T12:49:29.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:29.938+0000 7f6ee9418380 0 bucket checkpoint complete 2021-09-06T12:49:29.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:29.956 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:30.009 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:30.009 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:30.034 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.033+0000 7f385c0d8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:30.034 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.033+0000 7f385c0d8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:30.034 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.033+0000 7f385c0d8380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:30.034 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.033+0000 7f385c0d8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:30.066 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.065+0000 7f385c0d8380 1 bucket sync caught up with source: 2021-09-06T12:49:30.066 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000004.141.9, , , , , ] 2021-09-06T12:49:30.066 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000004.141.9, , , , , ] 2021-09-06T12:49:30.067 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:30.065+0000 7f385c0d8380 0 bucket checkpoint complete 2021-09-06T12:49:30.082 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone2, test-zone1} 2021-09-06T12:49:30.088 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:30.135 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.135 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:30.139 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.302 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone2, test-zone1} 2021-09-06T12:49:30.302 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone3, test-zone1} 2021-09-06T12:49:30.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:30.355 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.355 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:30.359 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.526 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone3, test-zone1} 2021-09-06T12:49:30.526 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone3, test-zone2} 2021-09-06T12:49:30.532 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:30.579 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.579 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:30.582 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:30.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:30.746 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone3, test-zone2} 2021-09-06T12:49:32.187 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:32.188 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:32.189 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:32.239 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:32.240 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:32.263 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:32.262+0000 7f0336c9a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:32.263 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:32.262+0000 7f0336c9a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:32.264 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:32.262+0000 7f0336c9a380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:32.264 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:32.262+0000 7f0336c9a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:32.296 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:32.295+0000 7f0336c9a380 1 waiting for incremental sync to catch up: 2021-09-06T12:49:32.296 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000056.137.5, , ] 2021-09-06T12:49:32.296 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000058.141.8, , ] 2021-09-06T12:49:37.298 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.297+0000 7f0336c9a380 1 bucket sync caught up with source: 2021-09-06T12:49:37.298 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000062.149.9, , ] 2021-09-06T12:49:37.299 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000058.141.8, , ] 2021-09-06T12:49:37.299 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.297+0000 7f0336c9a380 0 bucket checkpoint complete 2021-09-06T12:49:37.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.323 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:37.380 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:37.380 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:37.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.406+0000 7ff679dee380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:37.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.406+0000 7ff679dee380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:37.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.406+0000 7ff679dee380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:37.407 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.406+0000 7ff679dee380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:37.454 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.453+0000 7ff679dee380 1 bucket sync caught up with source: 2021-09-06T12:49:37.454 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000062.149.9, , ] 2021-09-06T12:49:37.454 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000062.149.9, , ] 2021-09-06T12:49:37.454 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.453+0000 7ff679dee380 0 bucket checkpoint complete 2021-09-06T12:49:37.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.471 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:37.528 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:37.528 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:37.554 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.554+0000 7fa5e60a5380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:37.555 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.554+0000 7fa5e60a5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:37.555 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.554+0000 7fa5e60a5380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:49:37.555 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.554+0000 7fa5e60a5380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:37.589 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.588+0000 7fa5e60a5380 1 bucket sync caught up with source: 2021-09-06T12:49:37.589 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000006.17.9, , , 00000000062.149.9, , ] 2021-09-06T12:49:37.589 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000006.17.9, , , 00000000062.149.9, , ] 2021-09-06T12:49:37.589 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:37.588+0000 7fa5e60a5380 0 bucket checkpoint complete 2021-09-06T12:49:37.605 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.607 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:37.659 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:37.659 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:37.684 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.683+0000 7f102fed1380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:37.684 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.683+0000 7f102fed1380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:37.684 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.683+0000 7f102fed1380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:37.684 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.683+0000 7f102fed1380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:37.716 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.716+0000 7f102fed1380 1 bucket sync caught up with source: 2021-09-06T12:49:37.716 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.337.9, , , 00000000061.426.9, , ] 2021-09-06T12:49:37.717 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.337.9, , , 00000000061.426.9, , ] 2021-09-06T12:49:37.717 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.716+0000 7f102fed1380 0 bucket checkpoint complete 2021-09-06T12:49:37.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.733 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:37.783 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:37.783 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:37.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.807+0000 7ff833d05380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:37.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.807+0000 7ff833d05380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:37.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.807+0000 7ff833d05380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:49:37.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.807+0000 7ff833d05380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:37.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.849+0000 7ff833d05380 1 bucket sync caught up with source: 2021-09-06T12:49:37.849 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.337.9, , , 00000000061.426.9, , ] 2021-09-06T12:49:37.850 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.337.9, , , 00000000061.426.9, , ] 2021-09-06T12:49:37.850 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.849+0000 7ff833d05380 0 bucket checkpoint complete 2021-09-06T12:49:37.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:37.869 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:37.918 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:37.919 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:37.941 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.941+0000 7fda00ead380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:37.942 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.941+0000 7fda00ead380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:37.942 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.941+0000 7fda00ead380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:49:37.942 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.941+0000 7fda00ead380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:37.982 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.981+0000 7fda00ead380 1 bucket sync caught up with source: 2021-09-06T12:49:37.982 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , 00000000004.141.9, , , 00000000053.335.9, , ] 2021-09-06T12:49:37.982 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , 00000000004.141.9, , , 00000000053.335.9, , ] 2021-09-06T12:49:37.982 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:49:37.981+0000 7fda00ead380 0 bucket checkpoint complete 2021-09-06T12:49:38.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:38.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-19', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:49:38.001 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-19 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:49:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:49:38.054 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:49:38.079 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.078+0000 7f9893fd7380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:49:38.079 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.078+0000 7f9893fd7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:49:38.080 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.078+0000 7f9893fd7380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:49:38.080 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.078+0000 7f9893fd7380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:49:38.112 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.112+0000 7f9893fd7380 1 bucket sync caught up with source: 2021-09-06T12:49:38.112 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , 00000000004.141.9, , , 00000000053.335.9, , ] 2021-09-06T12:49:38.113 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , 00000000004.141.9, , , 00000000053.335.9, , ] 2021-09-06T12:49:38.113 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:49:38.112+0000 7f9893fd7380 0 bucket checkpoint complete 2021-09-06T12:49:38.128 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone2, test-zone1} 2021-09-06T12:49:38.134 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.140 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.141 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.142 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.143 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.144 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.145 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.146 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.147 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.148 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.149 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.150 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.151 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.152 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:38.157 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:38.157 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.158 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.163 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.164 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.165 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.166 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:38.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:38.187 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:38.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:38.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.597 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.798 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:38.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.463 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:39.966 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.116 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.314 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.365 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.616 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:40.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.066 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.117 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.570 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:41.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.334 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.385 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.494 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.641 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:42.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.094 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.254 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.400 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:43.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.774 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:44.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:45.890 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone2, test-zone1} 2021-09-06T12:49:45.891 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone3, test-zone1} 2021-09-06T12:49:45.897 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:45.903 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:45.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.906 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.908 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.909 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.910 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.915 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.915 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:45.920 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:45.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.932 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:45.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:45.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:46.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:46.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.478 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.582 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.839 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:46.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.498 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.594 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:47.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.614 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:48.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:49.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.126 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.282 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.534 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.786 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:50.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.543 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.639 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.794 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:51.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:52.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.566 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.662 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone3, test-zone1} 2021-09-06T12:49:53.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-19 zones={test-zone3, test-zone2} 2021-09-06T12:49:53.669 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:49:53.675 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:53.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.676 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.677 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.678 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.685 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.686 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.689 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:49:53.695 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-09-06T12:49:53.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.701 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.702 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.703 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.704 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.705 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.706 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.707 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.708 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:49:53.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:53.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:53.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-09-06T12:49:53.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:53.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:54.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.404 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.858 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:55.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.014 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:56.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.173 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.279 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.681 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:57.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.638 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:58.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.047 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.394 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.550 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:49:59.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.054 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.558 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.706 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.906 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:00.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:01.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:01.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:01.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:01.315 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:01.410 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-19 zones={test-zone3, test-zone2} 2021-09-06T12:50:01.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:01.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:01.412 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:01.462 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:01.463 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:01.539 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:01.539 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:01.539 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:01.540 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:01.540 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:01.540 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:01.540 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:01.540 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:01.541 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:01.541 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:01.541 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:01.541 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:01.541 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:01.542 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:01.542 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:01.542 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:01.542 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:01.543 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:01.544 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-06T12:50:01.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:01.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:01.545 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:01.598 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:01.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:01.669 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:01.670 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:01.670 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:01.670 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:01.671 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:01.671 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 4902060214887631374 2021-09-06T12:50:01.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:01.671 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:01.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:01.672 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:01.672 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:01.672 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:01.672 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:01.673 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:01.673 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:01.673 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:01.673 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2021-09-06T12:50:01.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.674 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:01.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:01.674 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:01.674 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:01.675 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:01.675 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:01.675 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:01.675 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:01.675 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:01.676 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:01.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:01.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:01.677 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:01.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:01.677 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:01.677 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T12:50:01.677 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:01.678 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:01.678 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:01.678 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.129247Z" 2021-09-06T12:50:01.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:01.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:01.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:01.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:01.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:01.680 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:01.680 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:01.680 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:01.680 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:01.681 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:01.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.681 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:01.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:01.682 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:01.682 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2021-09-06T12:50:01.682 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2021-09-06T12:50:01.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:01.683 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:01.684 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000001177 target=00000000000000000000:00000000000000001044 2021-09-06T12:50:06.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:06.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:06.686 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:06.739 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:06.740 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:06.812 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:06.812 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:06.812 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:06.813 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:06.813 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:06.813 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 4902060214887631374 2021-09-06T12:50:06.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:06.813 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:06.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:06.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:06.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:06.814 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:06.814 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:06.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:06.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:06.815 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:06.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2021-09-06T12:50:06.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:06.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:06.816 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:06.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:06.817 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:06.817 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:06.817 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:06.817 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:06.817 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:06.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:06.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:06.818 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:06.818 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:06.819 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:06.819 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:06.819 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001044", 2021-09-06T12:50:06.819 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:06.819 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:06.820 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:06.820 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.129247Z" 2021-09-06T12:50:06.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.820 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:06.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:06.821 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:06.821 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:06.821 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:06.821 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:06.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.823 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:06.823 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:06.823 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:06.823 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2021-09-06T12:50:06.824 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2021-09-06T12:50:06.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:06.824 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:06.825 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=00000000000000000000:00000000000000001177 target=00000000000000000000:00000000000000001044 2021-09-06T12:50:11.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:11.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:11.827 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:11.881 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:11.881 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:11.966 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:11.966 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 4902060214887631374 2021-09-06T12:50:11.967 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:11.968 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:11.969 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2021-09-06T12:50:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:11.970 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:11.971 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:11.973 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:11.974 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:11.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:11.976 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.977 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2021-09-06T12:50:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2021-09-06T12:50:11.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:11.979 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:11.980 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-09-06T12:50:11.980 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:11.981 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:11.981 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.031 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.032 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.121 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:12.121 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.121 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:12.121 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:12.122 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.122 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.122 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.122 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:12.123 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.123 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.123 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.123 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:12.124 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.124 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.124 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.124 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:12.124 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.125 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:12.126 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-06T12:50:12.126 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.127 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.173 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.174 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.237 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:12.237 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:12.237 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:12.238 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:12.238 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:12.238 INFO:teuthology.orchestra.run.smithi041.stdout: "instance_id": 5446333851634123759 2021-09-06T12:50:12.238 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.239 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:12.239 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.239 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:12.239 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.239 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.240 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:12.240 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.240 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.240 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.240 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000" 2021-09-06T12:50:12.241 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.241 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.241 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.241 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:12.242 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.242 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.242 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.242 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.242 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.243 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.243 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:12.243 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.243 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.243 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.244 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:12.244 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.244 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.244 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.245 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.245 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.245 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.245 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:12.245 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.246 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.246 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.246 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:12.246 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.247 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.247 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.247 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.247 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.247 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.248 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:12.248 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.248 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.248 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:12.248 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.249 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:12.249 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 0, 2021-09-06T12:50:12.249 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 0 2021-09-06T12:50:12.249 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.249 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:12.250 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-09-06T12:50:12.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.252 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.294 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.294 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.376 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:12.376 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.376 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:12.376 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:12.377 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.377 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.377 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.377 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:12.377 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.378 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.378 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.378 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:12.378 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.379 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.379 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.379 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:12.379 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.379 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:12.380 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-06T12:50:12.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.382 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.434 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.434 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.503 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:12.503 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:12.503 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:12.503 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 18370994944478923316 2021-09-06T12:50:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.504 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.505 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:12.506 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:12.506 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:12.506 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:12.506 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2021-09-06T12:50:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:12.508 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:10.124588Z" 2021-09-06T12:50:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.510 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:54.135976Z" 2021-09-06T12:50:12.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.512 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:12.512 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:12.513 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.513 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.513 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:12.513 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:12.513 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:12.514 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.131956Z" 2021-09-06T12:50:12.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.514 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:12.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.515 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:12.515 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2021-09-06T12:50:12.515 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2021-09-06T12:50:12.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.516 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:12.517 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-09-06T12:50:12.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.518 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.518 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.561 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.561 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.630 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2021-09-06T12:50:12.630 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.631 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:12.631 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2021-09-06T12:50:12.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.631 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.632 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:10.153561Z" 2021-09-06T12:50:12.632 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.632 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.632 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.633 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:54.159824Z" 2021-09-06T12:50:12.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.633 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.633 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:32.158061Z" 2021-09-06T12:50:12.634 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.634 INFO:teuthology.orchestra.run.smithi052.stdout:] 2021-09-06T12:50:12.635 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-06T12:50:12.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.636 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.683 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.683 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.765 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:12.766 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:12.766 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:12.766 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:12.766 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:12.767 INFO:teuthology.orchestra.run.smithi041.stdout: "instance_id": 1635460851874055804 2021-09-06T12:50:12.767 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.767 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:12.767 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.767 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:12.768 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.768 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.768 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:12.768 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.769 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.769 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.769 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000" 2021-09-06T12:50:12.769 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.769 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.770 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.770 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:12.770 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.770 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.770 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.771 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.771 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.771 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.771 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:10.153561Z" 2021-09-06T12:50:12.771 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.772 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.772 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.772 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:12.772 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.772 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.773 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.773 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.773 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.773 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.773 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:54.159824Z" 2021-09-06T12:50:12.774 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.774 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.774 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:12.774 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:12.775 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:12.775 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:12.775 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.775 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:12.775 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:12.776 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:12.776 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:32.158061Z" 2021-09-06T12:50:12.776 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.776 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.776 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:12.777 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:12.777 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:12.777 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 0, 2021-09-06T12:50:12.777 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 0 2021-09-06T12:50:12.777 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:12.778 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:12.779 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-09-06T12:50:12.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.780 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:12.893 INFO:teuthology.orchestra.run.smithi052.stdout:[ 2021-09-06T12:50:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "0.000000" 2021-09-06T12:50:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:10.153561Z" 2021-09-06T12:50:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:54.159824Z" 2021-09-06T12:50:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "last_update": "2021-09-06T12:49:32.158061Z" 2021-09-06T12:50:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:12.897 INFO:teuthology.orchestra.run.smithi052.stdout:] 2021-09-06T12:50:12.898 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-06T12:50:12.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:12.899 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:12.945 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:12.945 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.012 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:13.012 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:13.013 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:13.013 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:13.013 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:13.013 INFO:teuthology.orchestra.run.smithi041.stdout: "instance_id": 12755721265202804601 2021-09-06T12:50:13.013 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.014 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:13.014 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.014 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:13.014 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.015 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.015 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:13.015 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.015 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.015 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.016 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000" 2021-09-06T12:50:13.016 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.016 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.016 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.017 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:13.017 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.017 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.017 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:13.017 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.018 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.018 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.018 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:10.153561Z" 2021-09-06T12:50:13.018 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:54.159824Z" 2021-09-06T12:50:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:32.158061Z" 2021-09-06T12:50:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 0, 2021-09-06T12:50:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 0 2021-09-06T12:50:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.025 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:13.026 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-09-06T12:50:13.026 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.027 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.069 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.070 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.152 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:13.152 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.153 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:13.153 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:13.153 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.153 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.154 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:13.154 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:10.135783Z" 2021-09-06T12:50:13.154 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.154 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.154 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:13.155 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:54.146168Z" 2021-09-06T12:50:13.155 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.155 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.155 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:13.156 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:32.141766Z" 2021-09-06T12:50:13.156 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.156 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:13.157 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-06T12:50:13.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.158 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.243 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.244 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.320 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:13.320 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:13.321 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:13.321 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:13.321 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:13.321 INFO:teuthology.orchestra.run.smithi041.stdout: "instance_id": 13935018405271926927 2021-09-06T12:50:13.321 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.322 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:13.322 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.322 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:13.322 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.323 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.323 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:13.323 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.323 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.323 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.324 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000" 2021-09-06T12:50:13.324 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.324 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.324 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.324 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:13.325 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.325 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.325 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:13.325 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.326 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.326 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.326 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:10.135783Z" 2021-09-06T12:50:13.326 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.326 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.327 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.327 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:13.327 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.327 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.328 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:13.328 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.328 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.328 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.328 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:54.146168Z" 2021-09-06T12:50:13.329 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.329 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.329 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.329 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:13.330 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:13.330 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.330 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:13.330 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:13.330 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:13.331 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:13.331 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:49:32.141766Z" 2021-09-06T12:50:13.331 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.331 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.332 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:13.332 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.332 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:13.332 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 0, 2021-09-06T12:50:13.332 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 0 2021-09-06T12:50:13.333 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.333 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:13.334 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-09-06T12:50:13.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.335 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.378 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.378 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.451 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:13.451 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.451 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:13.452 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:13.452 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.452 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.452 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:13.452 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:10.135783Z" 2021-09-06T12:50:13.453 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.453 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.453 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:13.453 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:54.146168Z" 2021-09-06T12:50:13.453 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.454 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.454 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:13.454 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:49:32.141766Z" 2021-09-06T12:50:13.454 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.455 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:13.455 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-06T12:50:13.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.457 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.507 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.507 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.576 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:13.576 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:13.577 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:13.577 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:13.577 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:13.577 INFO:teuthology.orchestra.run.smithi052.stdout: "instance_id": 14724044375924680399 2021-09-06T12:50:13.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.578 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:13.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.578 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:13.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.578 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.579 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:13.579 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.579 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.579 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.579 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000" 2021-09-06T12:50:13.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.580 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:13.580 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.581 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.581 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001975", 2021-09-06T12:50:13.581 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.581 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.581 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.582 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:10.135783Z" 2021-09-06T12:50:13.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.582 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.582 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.582 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:13.583 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.583 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.583 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000001177", 2021-09-06T12:50:13.583 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.583 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.584 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.584 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:54.146168Z" 2021-09-06T12:50:13.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.584 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.584 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.585 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:13.585 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.585 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "incremental-sync", 2021-09-06T12:50:13.585 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "00000000000000000000:00000000000000003837", 2021-09-06T12:50:13.585 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.586 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.586 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.586 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:49:32.141766Z" 2021-09-06T12:50:13.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.587 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:13.587 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.587 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:13.587 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 0, 2021-09-06T12:50:13.587 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 0 2021-09-06T12:50:13.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.588 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:13.589 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-09-06T12:50:13.589 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-09-06T12:50:13.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.614 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.656 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.657 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.715 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:50:13.733 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:13.733 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.733 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:13.734 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:50:13.734 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.734 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.734 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:13.735 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:13.735 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.735 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.735 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:13.735 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:50:13.736 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:13.736 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:13.736 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932613.611566_92.1", 2021-09-06T12:50:13.736 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:50:13.611566Z" 2021-09-06T12:50:13.736 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:13.737 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:13.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:50:13.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:13.739 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:13.785 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:13.786 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:13.852 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:13.852 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:13.853 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:13.854 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:13.855 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:13.857 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:13.858 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:13.859 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.860 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.861 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932613.601617_89.1", 2021-09-06T12:50:13.862 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:13.601617Z", 2021-09-06T12:50:13.863 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:13.864 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:13.865 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:13.865 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:13.866 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932613.611566_92.1 target=1_1630932613.601617_89.1 2021-09-06T12:50:18.868 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:18.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:18.869 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:18.919 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:18.920 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:18.989 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932613.611566_92.1", 2021-09-06T12:50:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:13.611566Z", 2021-09-06T12:50:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.003 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:19.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:50:19.004 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:50:19.005 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:19.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:19.006 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:19.057 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:19.057 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:19.130 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:19.130 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:19.130 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:19.130 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:19.131 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:19.131 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:19.131 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:19.131 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:19.131 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:19.132 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:19.132 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:19.132 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:19.132 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:19.133 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:19.133 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:19.133 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:50:19.133 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:19.133 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:19.134 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:19.134 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.134 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:19.134 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:19.135 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:19.135 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:19.135 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:19.135 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:19.136 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:19.136 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:19.136 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:19.136 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:50:19.136 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:19.137 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.137 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:19.137 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:19.137 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:19.137 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:19.138 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:19.138 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:19.138 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:19.138 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:19.139 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:19.139 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:19.139 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:19.139 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.139 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:19.140 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:19.140 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:19.140 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:19.140 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:19.141 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932613.611566_92.1", 2021-09-06T12:50:19.141 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:19.141 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:19.141 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:19.141 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:50:13.611566Z", 2021-09-06T12:50:19.142 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:19.142 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.142 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.142 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:19.143 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:19.143 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:19.143 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:50:19.143 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:50:19.143 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:19.144 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:19.145 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:50:19.145 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:50:19.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:19.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:19.147 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:19.193 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:19.193 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:19.256 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:19.256 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:19.256 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:19.257 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:19.257 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:19.257 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:19.257 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.258 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:19.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:19.258 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:19.258 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:19.259 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:19.259 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:19.259 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:19.259 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:19.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:19.260 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:19.260 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:19.261 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:19.261 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:19.261 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:19.261 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:19.262 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:19.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:19.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:19.262 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:19.262 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:19.263 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:19.263 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:19.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:19.264 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:19.264 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:19.264 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:19.264 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:19.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:19.265 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.265 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.265 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.265 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:19.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:19.266 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:19.266 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:19.266 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932613.611566_92.1", 2021-09-06T12:50:19.266 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:19.267 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:19.267 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:19.267 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:13.611566Z", 2021-09-06T12:50:19.267 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:19.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.268 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:19.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:19.268 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:19.269 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:19.269 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:19.269 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:19.269 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:19.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:50:19.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:19.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:19.320 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:19.371 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:19.394 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:19.395+0000 7f449828f380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:19.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:19.395+0000 7f449828f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:19.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:19.395+0000 7f449828f380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:19.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:19.395+0000 7f449828f380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:19.428 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:19.429+0000 7f449828f380 1 waiting for incremental sync to catch up: 2021-09-06T12:50:19.429 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , , , ] 2021-09-06T12:50:19.429 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.431 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.430+0000 7f449828f380 1 bucket sync caught up with source: 2021-09-06T12:50:24.432 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.432 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.432 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.430+0000 7f449828f380 0 bucket checkpoint complete 2021-09-06T12:50:24.451 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.452 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:24.510 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:24.511 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:24.537 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.537+0000 7f4c20516380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:24.537 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.537+0000 7f4c20516380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:24.537 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.537+0000 7f4c20516380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:24.538 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.537+0000 7f4c20516380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:24.572 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.572+0000 7f4c20516380 1 bucket sync caught up with source: 2021-09-06T12:50:24.573 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.573 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.573 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.572+0000 7f4c20516380 0 bucket checkpoint complete 2021-09-06T12:50:24.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.593 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:24.650 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:24.650 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:24.676 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.676+0000 7f746020e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:24.676 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.676+0000 7f746020e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:24.676 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.676+0000 7f746020e380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:24.677 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.676+0000 7f746020e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:24.711 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.711+0000 7f746020e380 1 bucket sync caught up with source: 2021-09-06T12:50:24.711 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.712 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.6.5, , ] 2021-09-06T12:50:24.712 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:24.711+0000 7f746020e380 0 bucket checkpoint complete 2021-09-06T12:50:24.728 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.729 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:24.782 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:24.782 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:24.807 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.806+0000 7f2cdfca7380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:24.807 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.806+0000 7f2cdfca7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:24.807 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.806+0000 7f2cdfca7380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:24.807 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.806+0000 7f2cdfca7380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:24.854 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.854+0000 7f2cdfca7380 1 bucket sync caught up with source: 2021-09-06T12:50:24.855 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.92.5, , ] 2021-09-06T12:50:24.855 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.92.5, , ] 2021-09-06T12:50:24.855 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.854+0000 7f2cdfca7380 0 bucket checkpoint complete 2021-09-06T12:50:24.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:24.877 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:24.928 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:24.928 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:24.951 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.950+0000 7ffadbfe9380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:24.951 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.950+0000 7ffadbfe9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:24.951 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.950+0000 7ffadbfe9380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:24.951 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.950+0000 7ffadbfe9380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:24.983 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.982+0000 7ffadbfe9380 1 bucket sync caught up with source: 2021-09-06T12:50:24.983 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.92.5, , ] 2021-09-06T12:50:24.983 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.92.5, , ] 2021-09-06T12:50:24.984 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:24.982+0000 7ffadbfe9380 0 bucket checkpoint complete 2021-09-06T12:50:25.001 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:25.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:25.002 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:25.052 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:25.052 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:25.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.075+0000 7f1a80113380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:25.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.075+0000 7f1a80113380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:25.076 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.075+0000 7f1a80113380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:25.077 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.075+0000 7f1a80113380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:25.123 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.122+0000 7f1a80113380 1 bucket sync caught up with source: 2021-09-06T12:50:25.123 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.134.5, , ] 2021-09-06T12:50:25.124 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.134.5, , ] 2021-09-06T12:50:25.124 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:25.122+0000 7f1a80113380 0 bucket checkpoint complete 2021-09-06T12:50:25.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:25.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:25.146 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:25.204 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:25.204 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.229+0000 7f94a715d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.229+0000 7f94a715d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.229+0000 7f94a715d380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:25.230 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.229+0000 7f94a715d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:25.264 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.264+0000 7f94a715d380 1 bucket sync caught up with source: 2021-09-06T12:50:25.264 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.134.5, , ] 2021-09-06T12:50:25.265 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.134.5, , ] 2021-09-06T12:50:25.265 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:25.264+0000 7f94a715d380 0 bucket checkpoint complete 2021-09-06T12:50:25.281 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:25.287 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:25.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:25.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:25.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:25.504 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:25.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.551 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:25.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.718 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:25.718 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:25.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:25.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:25.775 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:25.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:25.938 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:25.945 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:25.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:25.946 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:25.988 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:25.989 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:26.041 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:50:26.064 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:26.064 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:26.065 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:26.065 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:50:26.065 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:26.065 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:26.065 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:26.066 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:26.066 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:26.066 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:26.066 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:26.066 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:50:26.067 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:26.067 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:26.067 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932625.944792_94.1", 2021-09-06T12:50:26.067 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:50:25.944792Z" 2021-09-06T12:50:26.067 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:26.068 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:26.069 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:50:26.069 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:26.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:26.070 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:26.117 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:26.117 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:26.184 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:26.184 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:26.185 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:26.185 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:26.185 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:26.185 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:26.186 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:26.186 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:26.186 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:26.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:26.186 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:26.187 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:26.187 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:26.187 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:26.187 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:26.188 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:26.188 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:26.188 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:26.188 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:26.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:26.189 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:26.190 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:26.190 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:26.190 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:26.190 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:26.190 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:26.191 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:26.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:26.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:26.191 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:26.192 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:26.192 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:26.192 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:26.192 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:26.192 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:26.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:26.193 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:26.193 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:26.193 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.193 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:26.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:26.194 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:26.194 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:26.194 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:26.194 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932625.941124_93.1", 2021-09-06T12:50:26.195 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:26.195 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:26.195 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:26.195 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:25.941124Z", 2021-09-06T12:50:26.195 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:26.196 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:26.197 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:26.197 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:26.197 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:26.198 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932625.944792_94.1 target=1_1630932625.941124_93.1 2021-09-06T12:50:31.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.201 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:31.251 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:31.252 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:31.322 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:31.322 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:31.322 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:31.323 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:31.323 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:31.323 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:31.323 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.324 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:31.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.324 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:31.324 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.325 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.325 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:31.325 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.325 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:31.325 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.326 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:31.326 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.326 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.327 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:31.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.327 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.327 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:31.328 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.328 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.328 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.328 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:31.328 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:31.329 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.330 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.332 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932625.944792_94.1", 2021-09-06T12:50:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.333 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:25.944792Z", 2021-09-06T12:50:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:31.334 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:31.335 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.335 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:31.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:50:31.337 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:50:31.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.338 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:31.388 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:31.388 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:31.461 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:31.461 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:31.461 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:31.461 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:31.462 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:31.462 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:31.462 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:31.462 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:31.462 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:31.463 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:31.463 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:31.463 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:31.463 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:31.464 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:31.464 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:31.464 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:50:31.464 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:31.464 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:31.465 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:31.465 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.465 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:31.465 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:31.465 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:31.466 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:31.466 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:31.466 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:31.466 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:31.467 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:31.467 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:31.467 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:50:31.467 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:31.467 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.468 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:31.468 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:31.468 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:31.468 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:31.468 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:31.469 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:31.469 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:31.469 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:31.469 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:31.470 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:31.470 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:31.470 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.470 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:31.470 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:31.471 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:31.471 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:31.471 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:31.471 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932625.944792_94.1", 2021-09-06T12:50:31.472 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:31.472 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:31.472 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:31.472 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:50:25.944792Z", 2021-09-06T12:50:31.472 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:31.473 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.473 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.473 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:31.473 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:31.473 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:31.474 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:50:31.474 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:50:31.474 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:31.474 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:31.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:50:31.476 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:50:31.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:31.477 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:31.523 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:31.524 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:31.587 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:31.588 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:31.588 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:31.588 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:31.588 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:31.589 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:31.589 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.589 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:31.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.590 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:31.590 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.590 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:31.591 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.591 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:31.591 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.591 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:31.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932625.944792_94.1", 2021-09-06T12:50:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:25.944792Z", 2021-09-06T12:50:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:31.601 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:31.602 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:50:31.655 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.656 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:31.706 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:31.707 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:31.732 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.732+0000 7f9e62226380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:31.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.732+0000 7f9e62226380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:31.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.732+0000 7f9e62226380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:31.733 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.732+0000 7f9e62226380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:31.785 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.784+0000 7f9e62226380 1 bucket sync caught up with source: 2021-09-06T12:50:31.785 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:31.785 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:31.785 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:31.784+0000 7f9e62226380 0 bucket checkpoint complete 2021-09-06T12:50:31.801 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.802 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:31.866 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:31.866 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:31.891 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.890+0000 7f7ed1f36380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:31.891 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.890+0000 7f7ed1f36380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:31.891 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.890+0000 7f7ed1f36380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:31.892 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.890+0000 7f7ed1f36380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:31.932 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.932+0000 7f7ed1f36380 1 bucket sync caught up with source: 2021-09-06T12:50:31.932 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:31.932 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:31.933 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:31.932+0000 7f7ed1f36380 0 bucket checkpoint complete 2021-09-06T12:50:31.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:31.949 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:32.001 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:32.025 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.025+0000 7f7397baf380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:32.025 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.025+0000 7f7397baf380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:32.026 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.025+0000 7f7397baf380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:32.026 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.025+0000 7f7397baf380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:32.057 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.057+0000 7f7397baf380 1 bucket sync caught up with source: 2021-09-06T12:50:32.057 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:32.058 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.9.12, , ] 2021-09-06T12:50:32.058 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.057+0000 7f7397baf380 0 bucket checkpoint complete 2021-09-06T12:50:32.073 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.074 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:32.125 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:32.126 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:32.150 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.150+0000 7faee58f0380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:32.151 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.150+0000 7faee58f0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:32.151 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.150+0000 7faee58f0380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:32.151 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.150+0000 7faee58f0380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:32.195 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.195+0000 7faee58f0380 1 bucket sync caught up with source: 2021-09-06T12:50:32.195 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000004.107.9, , ] 2021-09-06T12:50:32.196 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000004.107.9, , ] 2021-09-06T12:50:32.196 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.195+0000 7faee58f0380 0 bucket checkpoint complete 2021-09-06T12:50:32.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.213 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:32.263 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:32.263 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:32.287 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.286+0000 7efe5a49e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:32.287 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.286+0000 7efe5a49e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:32.287 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.286+0000 7efe5a49e380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:32.287 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.286+0000 7efe5a49e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:32.319 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.319+0000 7efe5a49e380 1 bucket sync caught up with source: 2021-09-06T12:50:32.319 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000004.107.9, , ] 2021-09-06T12:50:32.320 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000004.107.9, , ] 2021-09-06T12:50:32.320 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.319+0000 7efe5a49e380 0 bucket checkpoint complete 2021-09-06T12:50:32.335 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.336 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:32.386 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:32.386 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:32.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.409+0000 7f1f56168380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:32.409 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.409+0000 7f1f56168380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:32.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.409+0000 7f1f56168380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:32.410 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.409+0000 7f1f56168380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:32.445 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.444+0000 7f1f56168380 1 bucket sync caught up with source: 2021-09-06T12:50:32.445 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000004.149.9, , ] 2021-09-06T12:50:32.445 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000004.149.9, , ] 2021-09-06T12:50:32.445 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:32.444+0000 7f1f56168380 0 bucket checkpoint complete 2021-09-06T12:50:32.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:32.462 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:32.515 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:32.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:32.542 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.541+0000 7f67a851d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:32.542 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.541+0000 7f67a851d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:32.542 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.541+0000 7f67a851d380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:32.542 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.541+0000 7f67a851d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:32.575 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.575+0000 7f67a851d380 1 bucket sync caught up with source: 2021-09-06T12:50:32.575 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000004.149.9, , ] 2021-09-06T12:50:32.576 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000004.149.9, , ] 2021-09-06T12:50:32.576 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:32.575+0000 7f67a851d380 0 bucket checkpoint complete 2021-09-06T12:50:32.592 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:32.597 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:32.643 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.644 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.644 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:32.647 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.648 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.882 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:32.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:32.934 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:32.940 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:32.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.987 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.988 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:32.991 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:32.992 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:33.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.053 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.278 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:33.278 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:33.284 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:33.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:33.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:33.332 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:33.335 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:33.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:33.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:33.618 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:33.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:33.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:33.626 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:33.668 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:33.669 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:33.723 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:50:33.746 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:33.746 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:33.746 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:33.746 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:50:33.746 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:33.747 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:33.747 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:33.747 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:33.747 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:33.748 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:33.748 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:33.748 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:50:33.748 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:33.748 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:33.749 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932633.625060_96.1", 2021-09-06T12:50:33.749 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:50:33.625060Z" 2021-09-06T12:50:33.749 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:33.749 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:33.750 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:50:33.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:33.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:33.752 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:33.800 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:33.801 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:33.869 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:33.870 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:33.870 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:33.870 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:33.870 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:33.871 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:33.872 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:33.873 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:33.873 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:33.873 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:33.873 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:33.873 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:33.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:33.874 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:33.874 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:33.874 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:33.875 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:33.875 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:33.875 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:33.875 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:33.875 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:33.876 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:33.876 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:33.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:33.876 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:33.877 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:33.877 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:33.877 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:33.877 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:33.877 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:33.878 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:33.878 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:33.878 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:33.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:33.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:33.879 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:33.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:33.879 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:33.879 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932633.621299_95.1", 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:33.621299Z", 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:33.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.881 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:33.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:33.881 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:33.881 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:33.882 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:33.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:33.882 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:33.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932633.625060_96.1 target=1_1630932633.621299_95.1 2021-09-06T12:50:38.885 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:38.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:38.886 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:38.936 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:38.936 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.007 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.016 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:39.016 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.016 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.016 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.017 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:39.017 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.017 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.017 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932633.625060_96.1", 2021-09-06T12:50:39.018 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.018 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.018 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.018 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:33.625060Z", 2021-09-06T12:50:39.019 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.019 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:39.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.020 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:39.020 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:39.020 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:39.020 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.020 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:39.021 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:50:39.022 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:50:39.022 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:39.023 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:39.023 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:39.074 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.074 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.148 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:39.149 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:39.149 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:39.149 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:39.149 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:39.149 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:39.150 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:39.150 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:39.150 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:39.150 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:39.151 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:39.151 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:39.151 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:39.151 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:39.151 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:39.152 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:50:39.152 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:39.152 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:39.152 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:39.152 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.153 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:39.153 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:39.153 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:39.153 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:39.153 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:39.154 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:39.154 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:39.154 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:39.154 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:39.154 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:50:39.155 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:39.155 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.155 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:39.155 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:39.155 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:39.156 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:39.156 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:39.156 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:39.156 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:39.157 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:39.157 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:39.157 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:39.157 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:39.157 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.158 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:39.158 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:39.158 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:39.158 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:39.158 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:39.159 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932633.625060_96.1", 2021-09-06T12:50:39.159 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:39.159 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:39.159 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:39.159 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:50:33.625060Z", 2021-09-06T12:50:39.160 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:39.160 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.160 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.160 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:39.160 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:39.161 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:39.161 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:50:39.161 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:50:39.161 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:39.161 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:39.163 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:50:39.163 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:50:39.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:39.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:39.164 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:39.212 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.212 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.276 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:39.276 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:39.277 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.278 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.279 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.280 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.281 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.282 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:39.284 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:39.286 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932633.625060_96.1", 2021-09-06T12:50:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:39.287 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:33.625060Z", 2021-09-06T12:50:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:39.289 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:39.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:39.290 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:39.291 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:50:39.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.340 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:39.393 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.393 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.417 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.417+0000 7f66a0b2a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.417+0000 7f66a0b2a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.417+0000 7f66a0b2a380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:39.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.417+0000 7f66a0b2a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:39.451 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.451+0000 7f66a0b2a380 1 bucket sync caught up with source: 2021-09-06T12:50:39.452 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.452 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.452 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.451+0000 7f66a0b2a380 0 bucket checkpoint complete 2021-09-06T12:50:39.468 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.469 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:39.521 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.521 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.543 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.543+0000 7f6b5c2bc380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:39.544 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.543+0000 7f6b5c2bc380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:39.544 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.543+0000 7f6b5c2bc380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:39.544 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.543+0000 7f6b5c2bc380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:39.588 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.588+0000 7f6b5c2bc380 1 bucket sync caught up with source: 2021-09-06T12:50:39.589 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.589 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.589 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.588+0000 7f6b5c2bc380 0 bucket checkpoint complete 2021-09-06T12:50:39.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.605 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.605 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:39.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.680 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.680+0000 7f78e290a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:39.680 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.680+0000 7f78e290a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:39.681 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.680+0000 7f78e290a380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:39.681 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.680+0000 7f78e290a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:39.712 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.712+0000 7f78e290a380 1 bucket sync caught up with source: 2021-09-06T12:50:39.713 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.713 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000005.13.17, , ] 2021-09-06T12:50:39.713 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:39.712+0000 7f78e290a380 0 bucket checkpoint complete 2021-09-06T12:50:39.730 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.731 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.731 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:39.783 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.783 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.808+0000 7f0ddc0e9380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:39.808 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.808+0000 7f0ddc0e9380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:39.809 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.808+0000 7f0ddc0e9380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:39.809 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.808+0000 7f0ddc0e9380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:39.861 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.860+0000 7f0ddc0e9380 1 bucket sync caught up with source: 2021-09-06T12:50:39.861 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000007.123.14, , ] 2021-09-06T12:50:39.861 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000007.123.14, , ] 2021-09-06T12:50:39.861 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.860+0000 7f0ddc0e9380 0 bucket checkpoint complete 2021-09-06T12:50:39.886 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.886 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:39.887 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:39.937 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:39.937 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:39.962 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.961+0000 7f5f40f98380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:39.962 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.961+0000 7f5f40f98380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:39.962 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.961+0000 7f5f40f98380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:39.963 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:39.961+0000 7f5f40f98380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:40.005 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.004+0000 7f5f40f98380 1 bucket sync caught up with source: 2021-09-06T12:50:40.005 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000007.123.14, , ] 2021-09-06T12:50:40.006 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000007.123.14, , ] 2021-09-06T12:50:40.006 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.004+0000 7f5f40f98380 0 bucket checkpoint complete 2021-09-06T12:50:40.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:40.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:40.028 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:40.078 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:40.078 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:40.101 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.101+0000 7efc4f023380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:40.102 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.101+0000 7efc4f023380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:40.102 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.101+0000 7efc4f023380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:40.102 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.101+0000 7efc4f023380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:40.146 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.145+0000 7efc4f023380 1 bucket sync caught up with source: 2021-09-06T12:50:40.146 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000007.165.14, , ] 2021-09-06T12:50:40.146 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000007.165.14, , ] 2021-09-06T12:50:40.147 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:40.145+0000 7efc4f023380 0 bucket checkpoint complete 2021-09-06T12:50:40.165 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:40.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-20', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:40.166 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-20 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:40.219 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:40.219 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:40.244 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.243+0000 7f3fbfd75380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:40.244 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.243+0000 7f3fbfd75380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:40.244 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.243+0000 7f3fbfd75380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:40.244 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.243+0000 7f3fbfd75380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:40.277 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.277+0000 7f3fbfd75380 1 bucket sync caught up with source: 2021-09-06T12:50:40.277 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000007.165.14, , ] 2021-09-06T12:50:40.278 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000007.165.14, , ] 2021-09-06T12:50:40.278 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:40.277+0000 7f3fbfd75380 0 bucket checkpoint complete 2021-09-06T12:50:40.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:40.299 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:40.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.343 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.344 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:40.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.402 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.410 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.581 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.634 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone2, test-zone1} 2021-09-06T12:50:40.634 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:40.640 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:40.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:40.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:40.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:40.970 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone1} 2021-09-06T12:50:40.970 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:40.976 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:41.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:41.023 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:41.024 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:41.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:41.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:41.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.086 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.201 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:41.310 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-20 zones={test-zone3, test-zone2} 2021-09-06T12:50:41.311 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-09-06T12:50:41.311 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-21 2021-09-06T12:50:41.339 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-22 2021-09-06T12:50:41.379 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-23 2021-09-06T12:50:41.439 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:41.440 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:41.440 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:41.483 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:41.483 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:41.540 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:50:41.565 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:41.565 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:41.565 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:41.565 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:50:41.566 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:41.566 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:41.566 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:41.566 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:41.567 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:41.567 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:41.567 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:41.567 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:50:41.567 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:41.568 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:41.568 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932641.436881_115.1", 2021-09-06T12:50:41.568 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:50:41.436881Z" 2021-09-06T12:50:41.568 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:41.569 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:41.570 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:50:41.570 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:41.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:41.571 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:41.620 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:41.620 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:41.691 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:41.692 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:41.692 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:41.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:41.692 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:41.692 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:41.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:41.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:41.693 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:41.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:41.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:41.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:41.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:41.694 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:41.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:41.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:41.695 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:41.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:41.695 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:41.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:41.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:41.696 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:41.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:41.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:41.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:41.697 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:41.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:41.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:41.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:41.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:41.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.698 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:41.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:41.699 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:41.699 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:41.699 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:41.700 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:41.700 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:41.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:41.700 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:41.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:41.701 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:41.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:41.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:41.702 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:41.702 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:41.702 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:41.702 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932641.323335_97.1", 2021-09-06T12:50:41.702 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:41.703 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:41.703 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:41.703 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:41.323335Z", 2021-09-06T12:50:41.703 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:41.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.704 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:41.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:41.704 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:41.705 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:41.705 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:41.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:41.705 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:41.706 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932641.436881_115.1 target=1_1630932641.323335_97.1 2021-09-06T12:50:46.707 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.709 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:46.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:46.759 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:46.830 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:46.830 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:46.830 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:46.830 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:46.831 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:46.831 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:46.831 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:46.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:46.831 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:46.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:46.832 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:46.832 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:46.832 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:46.833 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:46.833 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:46.833 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:46.833 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:46.833 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:46.834 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:46.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.834 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:46.834 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:46.834 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932641.436881_115.1", 2021-09-06T12:50:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:41.436881Z", 2021-09-06T12:50:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:46.843 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:46.843 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:46.844 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:50:46.844 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:50:46.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.846 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:46.895 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:46.895 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:46.978 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:46.978 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:46.979 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:46.979 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:46.979 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:46.979 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:46.979 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:46.980 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:46.980 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:46.980 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:46.980 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:46.981 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:46.981 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:46.981 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:46.982 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:46.982 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:50:46.982 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:46.983 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:46.983 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:46.983 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.983 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:46.983 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:46.984 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:46.984 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:46.984 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:46.984 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:46.984 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:46.985 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:46.985 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:46.985 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:50:46.985 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:46.985 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.986 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:46.986 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:46.986 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:46.986 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:46.986 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:46.987 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:46.987 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:46.987 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:46.987 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:46.987 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:46.988 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:46.988 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.988 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:46.988 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:46.988 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:46.989 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:46.989 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:46.989 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932641.436881_115.1", 2021-09-06T12:50:46.989 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:46.989 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:46.990 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:46.990 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:50:41.436881Z", 2021-09-06T12:50:46.990 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:46.990 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.990 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.991 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:46.991 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:46.991 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:46.991 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:50:46.991 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:50:46.992 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:46.992 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:46.993 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:50:46.994 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:50:46.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:46.995 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:47.041 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:47.104 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:47.105 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:47.105 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:47.105 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:47.105 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:47.106 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:47.106 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:47.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:47.106 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:47.106 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:47.107 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:47.107 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:47.107 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:47.107 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:47.107 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:47.108 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:47.108 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:47.108 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:47.108 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:47.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:47.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:47.109 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:47.109 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:47.109 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:47.110 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:47.110 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:47.110 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:47.110 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:47.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:47.111 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:47.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:47.112 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:47.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:47.113 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:47.113 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:47.113 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:47.114 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:47.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:47.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:47.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:47.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:47.115 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:47.115 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932641.436881_115.1", 2021-09-06T12:50:47.115 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:47.116 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:47.116 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:47.116 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:41.436881Z", 2021-09-06T12:50:47.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:47.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.117 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:47.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:47.117 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:47.117 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:47.118 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:47.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:47.118 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:47.119 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:50:47.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.299 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:47.353 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:47.353 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:47.376 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.376+0000 7ff97445b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:47.376 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.376+0000 7ff97445b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:47.376 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.376+0000 7ff97445b380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:47.377 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.376+0000 7ff97445b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:47.410 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.410+0000 7ff97445b380 1 bucket sync caught up with source: 2021-09-06T12:50:47.411 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.411 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.411 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.410+0000 7ff97445b380 0 bucket checkpoint complete 2021-09-06T12:50:47.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.432 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:47.496 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:47.496 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:47.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.520+0000 7f941d2e8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:47.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.520+0000 7f941d2e8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:47.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.520+0000 7f941d2e8380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:47.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.520+0000 7f941d2e8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:47.569 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.568+0000 7f941d2e8380 1 bucket sync caught up with source: 2021-09-06T12:50:47.569 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.570 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.570 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.568+0000 7f941d2e8380 0 bucket checkpoint complete 2021-09-06T12:50:47.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.593 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:47.649 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:47.650 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:47.673 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.672+0000 7f7f03c2d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:47.673 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.672+0000 7f7f03c2d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:47.673 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.672+0000 7f7f03c2d380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:47.674 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.672+0000 7f7f03c2d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:47.706 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.706+0000 7f7f03c2d380 1 bucket sync caught up with source: 2021-09-06T12:50:47.707 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.707 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.15.11, , ] 2021-09-06T12:50:47.707 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:47.706+0000 7f7f03c2d380 0 bucket checkpoint complete 2021-09-06T12:50:47.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.724 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:47.780 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:47.780 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:47.802 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.802+0000 7fa9ca082380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:47.803 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.802+0000 7fa9ca082380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:47.803 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.802+0000 7fa9ca082380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:47.803 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.802+0000 7fa9ca082380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:47.848 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.848+0000 7fa9ca082380 1 bucket sync caught up with source: 2021-09-06T12:50:47.849 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.167.5, , ] 2021-09-06T12:50:47.849 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.167.5, , ] 2021-09-06T12:50:47.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.848+0000 7fa9ca082380 0 bucket checkpoint complete 2021-09-06T12:50:47.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:47.872 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:47.922 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:48.012 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:48.013 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.944+0000 7f720da80380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:48.013 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.944+0000 7f720da80380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:48.014 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.944+0000 7f720da80380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:48.014 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.944+0000 7f720da80380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:48.014 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.990+0000 7f720da80380 1 bucket sync caught up with source: 2021-09-06T12:50:48.014 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.167.5, , ] 2021-09-06T12:50:48.015 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.167.5, , ] 2021-09-06T12:50:48.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:47.990+0000 7f720da80380 0 bucket checkpoint complete 2021-09-06T12:50:48.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.017 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:48.067 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:48.067 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:48.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.090+0000 7fabe2552380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:48.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.090+0000 7fabe2552380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:48.091 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.090+0000 7fabe2552380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:48.092 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.090+0000 7fabe2552380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:48.125 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.125+0000 7fabe2552380 1 bucket sync caught up with source: 2021-09-06T12:50:48.126 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.249.5, , ] 2021-09-06T12:50:48.126 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.249.5, , ] 2021-09-06T12:50:48.126 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:48.125+0000 7fabe2552380 0 bucket checkpoint complete 2021-09-06T12:50:48.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-21', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.142 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-21 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:48.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:48.196 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:48.220 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.220+0000 7f9bb3db4380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:48.221 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.220+0000 7f9bb3db4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:48.221 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.220+0000 7f9bb3db4380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:48.221 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.220+0000 7f9bb3db4380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:48.253 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.253+0000 7f9bb3db4380 1 bucket sync caught up with source: 2021-09-06T12:50:48.254 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.249.5, , ] 2021-09-06T12:50:48.254 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.249.5, , ] 2021-09-06T12:50:48.254 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.253+0000 7f9bb3db4380 0 bucket checkpoint complete 2021-09-06T12:50:48.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-21 zones={test-zone2, test-zone1} 2021-09-06T12:50:48.276 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:48.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:48.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-21 zones={test-zone2, test-zone1} 2021-09-06T12:50:48.446 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-21 zones={test-zone3, test-zone1} 2021-09-06T12:50:48.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:48.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.567 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.622 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-21 zones={test-zone3, test-zone1} 2021-09-06T12:50:48.622 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-21 zones={test-zone3, test-zone2} 2021-09-06T12:50:48.628 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:48.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.676 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:48.679 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.680 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:48.730 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.742 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:48.802 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-21 zones={test-zone3, test-zone2} 2021-09-06T12:50:48.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.803 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:48.854 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:48.854 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:48.879 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.879+0000 7f3065a93380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:48.879 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.879+0000 7f3065a93380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:48.879 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.879+0000 7f3065a93380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:48.880 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.879+0000 7f3065a93380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.912+0000 7f3065a93380 1 bucket sync caught up with source: 2021-09-06T12:50:48.913 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:48.913 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:48.913 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:48.912+0000 7f3065a93380 0 bucket checkpoint complete 2021-09-06T12:50:48.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:48.930 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:48.982 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:48.983 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.007 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.008+0000 7fccc94b4380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.007 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.008+0000 7fccc94b4380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.007 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.008+0000 7fccc94b4380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:49.008 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.008+0000 7fccc94b4380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.047+0000 7fccc94b4380 1 bucket sync caught up with source: 2021-09-06T12:50:49.047 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:49.047 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:49.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.047+0000 7fccc94b4380 0 bucket checkpoint complete 2021-09-06T12:50:49.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.063 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:49.114 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:49.114 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.136+0000 7f12b46ad380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.136+0000 7f12b46ad380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.136+0000 7f12b46ad380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:49.137 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.136+0000 7f12b46ad380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.168 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.167+0000 7f12b46ad380 1 bucket sync caught up with source: 2021-09-06T12:50:49.168 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:49.168 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.15.5, , ] 2021-09-06T12:50:49.168 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.167+0000 7f12b46ad380 0 bucket checkpoint complete 2021-09-06T12:50:49.183 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.185 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:49.236 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:49.236 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.260 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.261+0000 7f97ff2d8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.261 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.261+0000 7f97ff2d8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.261 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.261+0000 7f97ff2d8380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:49.261 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.261+0000 7f97ff2d8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.293 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.294+0000 7f97ff2d8380 1 bucket sync caught up with source: 2021-09-06T12:50:49.294 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.281.11, , ] 2021-09-06T12:50:49.294 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.281.11, , ] 2021-09-06T12:50:49.294 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.294+0000 7f97ff2d8380 0 bucket checkpoint complete 2021-09-06T12:50:49.310 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.310 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.311 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:49.361 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:49.362 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.386 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.387+0000 7f01afa82380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.387 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.387+0000 7f01afa82380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.387 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.387+0000 7f01afa82380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:49.387 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.387+0000 7f01afa82380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.430 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.430+0000 7f01afa82380 1 bucket sync caught up with source: 2021-09-06T12:50:49.430 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.281.11, , ] 2021-09-06T12:50:49.431 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.281.11, , ] 2021-09-06T12:50:49.431 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.430+0000 7f01afa82380 0 bucket checkpoint complete 2021-09-06T12:50:49.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.450 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:49.500 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:49.500 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.523 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.523+0000 7fa5006d2380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.523 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.523+0000 7fa5006d2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.523 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.523+0000 7fa5006d2380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:49.524 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.523+0000 7fa5006d2380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.573 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.573+0000 7fa5006d2380 1 bucket sync caught up with source: 2021-09-06T12:50:49.573 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.345.5, , ] 2021-09-06T12:50:49.573 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.345.5, , ] 2021-09-06T12:50:49.574 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:49.573+0000 7fa5006d2380 0 bucket checkpoint complete 2021-09-06T12:50:49.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-22', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:49.590 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-22 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:49.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:49.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:49.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.665+0000 7f53c43a0380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:49.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.665+0000 7f53c43a0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:49.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.665+0000 7f53c43a0380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:49.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.665+0000 7f53c43a0380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:49.699 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.698+0000 7f53c43a0380 1 bucket sync caught up with source: 2021-09-06T12:50:49.699 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.345.5, , ] 2021-09-06T12:50:49.699 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.345.5, , ] 2021-09-06T12:50:49.699 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:49.699+0000 7f53c43a0380 0 bucket checkpoint complete 2021-09-06T12:50:49.715 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-22 zones={test-zone2, test-zone1} 2021-09-06T12:50:49.722 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:49.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.767 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.768 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:49.771 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.772 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:49.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:49.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:49.898 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-22 zones={test-zone2, test-zone1} 2021-09-06T12:50:49.899 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-22 zones={test-zone3, test-zone1} 2021-09-06T12:50:49.904 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:49.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.952 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:49.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:49.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:50.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.073 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-22 zones={test-zone3, test-zone1} 2021-09-06T12:50:50.074 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-22 zones={test-zone3, test-zone2} 2021-09-06T12:50:50.080 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:50.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:50.127 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:50.128 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:50.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:50.132 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:50.182 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:50.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-22 zones={test-zone3, test-zone2} 2021-09-06T12:50:50.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.251 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.302 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.302 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.324 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.324+0000 7f5644a0a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.325 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.324+0000 7f5644a0a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.325 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.324+0000 7f5644a0a380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:50.325 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.324+0000 7f5644a0a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:50.356 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.356+0000 7f5644a0a380 1 bucket sync caught up with source: 2021-09-06T12:50:50.356 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.357 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.357 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.356+0000 7f5644a0a380 0 bucket checkpoint complete 2021-09-06T12:50:50.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.374 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.374 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.425 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.425 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.449 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.449+0000 7f8317b08380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.449 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.449+0000 7f8317b08380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.449 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.449+0000 7f8317b08380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:50.450 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.449+0000 7f8317b08380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:50.483 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.484+0000 7f8317b08380 1 bucket sync caught up with source: 2021-09-06T12:50:50.483 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.484 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.484 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.484+0000 7f8317b08380 0 bucket checkpoint complete 2021-09-06T12:50:50.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.501 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.552 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.552 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.577 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.576+0000 7fcdb83e7380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.577 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.576+0000 7fcdb83e7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.577 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.576+0000 7fcdb83e7380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:50.578 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.576+0000 7fcdb83e7380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:50.609 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.609+0000 7fcdb83e7380 1 bucket sync caught up with source: 2021-09-06T12:50:50.609 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.610 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000002.8.5, , ] 2021-09-06T12:50:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:50.609+0000 7fcdb83e7380 0 bucket checkpoint complete 2021-09-06T12:50:50.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.626 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.676 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.676 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.700 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.701+0000 7ff3ca61d380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.700 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.701+0000 7ff3ca61d380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.701 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.701+0000 7ff3ca61d380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:50.701 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.701+0000 7ff3ca61d380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:50.741 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.742+0000 7ff3ca61d380 1 bucket sync caught up with source: 2021-09-06T12:50:50.742 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.124.5, , ] 2021-09-06T12:50:50.742 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.124.5, , ] 2021-09-06T12:50:50.742 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.742+0000 7ff3ca61d380 0 bucket checkpoint complete 2021-09-06T12:50:50.764 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.765 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.815 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.815 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.839 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.840+0000 7f0f9572a380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.840 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.840+0000 7f0f9572a380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.840 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.840+0000 7f0f9572a380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:50.840 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.840+0000 7f0f9572a380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:50.886 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.887+0000 7f0f9572a380 1 bucket sync caught up with source: 2021-09-06T12:50:50.886 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000002.124.5, , ] 2021-09-06T12:50:50.887 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000002.124.5, , ] 2021-09-06T12:50:50.887 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.887+0000 7f0f9572a380 0 bucket checkpoint complete 2021-09-06T12:50:50.905 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:50.906 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:50.957 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:50.957 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:50.981 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.981+0000 7f36a14e5380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:50.981 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.981+0000 7f36a14e5380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:50.981 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.981+0000 7f36a14e5380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:50.981 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:50.981+0000 7f36a14e5380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:51.015 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:51.016+0000 7f36a14e5380 1 bucket sync caught up with source: 2021-09-06T12:50:51.015 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.156.11, , ] 2021-09-06T12:50:51.015 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.156.11, , ] 2021-09-06T12:50:51.016 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:51.016+0000 7f36a14e5380 0 bucket checkpoint complete 2021-09-06T12:50:51.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:51.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-23', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:51.032 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-23 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:51.084 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:51.084 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:51.107 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.107+0000 7f55b77af380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:51.108 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.107+0000 7f55b77af380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:51.108 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.107+0000 7f55b77af380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:51.108 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.107+0000 7f55b77af380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:51.140 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.139+0000 7f55b77af380 1 bucket sync caught up with source: 2021-09-06T12:50:51.140 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.156.11, , ] 2021-09-06T12:50:51.140 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.156.11, , ] 2021-09-06T12:50:51.140 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:51.139+0000 7f55b77af380 0 bucket checkpoint complete 2021-09-06T12:50:51.156 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-23 zones={test-zone2, test-zone1} 2021-09-06T12:50:51.162 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:51.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.207 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.208 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:51.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.258 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.325 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-23 zones={test-zone2, test-zone1} 2021-09-06T12:50:51.326 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-23 zones={test-zone3, test-zone1} 2021-09-06T12:50:51.331 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:51.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.375 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:51.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.379 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.498 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-23 zones={test-zone3, test-zone1} 2021-09-06T12:50:51.498 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-23 zones={test-zone3, test-zone2} 2021-09-06T12:50:51.504 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:51.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.555 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:51.559 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.560 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:51.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:51.678 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-23 zones={test-zone3, test-zone2} 2021-09-06T12:50:51.678 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-09-06T12:50:51.679 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-24 2021-09-06T12:50:51.701 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-25 2021-09-06T12:50:51.744 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-26 2021-09-06T12:50:51.829 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:51.829 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:51.830 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:51.876 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:51.877 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:51.933 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:50:51.955 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:50:51.955 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:51.955 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:51.955 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:50:51.956 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:51.956 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:51.956 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:51.956 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:50:51.956 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:51.957 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:51.957 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:51.957 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:50:51.957 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:51.957 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:51.958 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932651.826059_162.1", 2021-09-06T12:50:51.958 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:50:51.826059Z" 2021-09-06T12:50:51.958 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:51.958 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:50:51.959 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:50:51.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:51.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:51.961 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:52.004 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:52.067 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:52.068 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:52.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:52.070 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:52.071 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:52.072 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:52.073 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:52.074 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:52.075 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:52.076 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:52.077 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932651.689833_139.1", 2021-09-06T12:50:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:52.078 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:51.689833Z", 2021-09-06T12:50:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.079 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:52.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:52.081 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:52.082 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932651.826059_162.1 target=1_1630932651.689833_139.1 2021-09-06T12:50:57.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.084 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:57.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:57.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:57.205 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:57.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:57.208 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.208 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.208 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:57.209 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.209 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:57.209 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.209 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:57.209 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.210 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:57.210 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.211 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.211 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:57.211 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.211 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.211 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.212 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:57.212 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.212 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.213 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:57.213 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.213 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.213 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:57.213 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.214 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.214 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.214 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:57.214 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.215 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.215 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:57.215 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.215 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.216 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932651.826059_162.1", 2021-09-06T12:50:57.216 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.216 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.216 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.216 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:51.826059Z", 2021-09-06T12:50:57.217 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.217 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.217 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:57.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.218 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:57.218 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:57.218 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:57.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.218 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:57.219 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:50:57.220 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:50:57.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.221 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:57.272 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:57.272 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:57.357 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:50:57.357 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:50:57.357 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:50:57.358 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:50:57.358 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:50:57.358 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:57.358 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:57.358 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:57.359 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:50:57.359 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:57.359 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:50:57.359 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:57.359 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:57.360 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:57.360 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:57.360 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:50:57.360 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:57.360 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:57.361 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:57.361 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.361 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:57.361 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:57.361 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:50:57.362 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:57.362 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:57.362 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:50:57.362 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:50:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:50:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:50:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:50:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:50:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932651.826059_162.1", 2021-09-06T12:50:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:50:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:50:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:50:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:50:51.826059Z", 2021-09-06T12:50:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:50:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:50:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:50:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:50:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:50:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:50:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:50:57.370 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:50:57.371 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T12:50:57.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T12:50:57.371 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:50:57.372 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:50:57.418 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:57.418 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:57.482 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:50:57.482 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:50:57.482 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:50:57.483 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:50:57.483 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:50:57.483 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:50:57.483 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.484 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:50:57.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:50:57.484 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.485 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.485 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:50:57.485 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.485 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:50:57.485 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.486 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:50:57.486 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.487 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:50:57.487 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.487 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.487 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:50:57.487 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.488 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.488 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:50:57.488 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:50:57.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.489 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.490 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:50:57.490 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.490 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.490 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.490 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:50:57.491 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:50:57.491 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932651.826059_162.1", 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:50:57.492 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:50:57.493 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:50:51.826059Z", 2021-09-06T12:50:57.493 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:50:57.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.493 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:50:57.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:50:57.494 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:50:57.494 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:50:57.494 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:50:57.494 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:50:57.495 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:50:57.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-09-06T12:50:57.690 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.691 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:57.748 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:57.748 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:57.774 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.774+0000 7fd7f6b69380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:57.775 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.774+0000 7fd7f6b69380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:57.775 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.774+0000 7fd7f6b69380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:57.775 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.774+0000 7fd7f6b69380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:57.810 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.810+0000 7fd7f6b69380 1 bucket sync caught up with source: 2021-09-06T12:50:57.810 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:57.810 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:57.811 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:57.810+0000 7fd7f6b69380 0 bucket checkpoint complete 2021-09-06T12:50:57.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.828 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:57.891 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:57.891 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:57.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.917+0000 7fdecb65c380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:57.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.917+0000 7fdecb65c380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:57.917 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.917+0000 7fdecb65c380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:57.918 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.917+0000 7fdecb65c380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:57.965 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.965+0000 7fdecb65c380 1 bucket sync caught up with source: 2021-09-06T12:50:57.965 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:57.965 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:57.965 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:57.965+0000 7fdecb65c380 0 bucket checkpoint complete 2021-09-06T12:50:57.984 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.984 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:57.985 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:58.039 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:58.039 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:58.066 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.065+0000 7fb0e8679380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:58.066 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.065+0000 7fb0e8679380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:58.066 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.065+0000 7fb0e8679380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:58.067 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.065+0000 7fb0e8679380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:58.100 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.100+0000 7fb0e8679380 1 bucket sync caught up with source: 2021-09-06T12:50:58.100 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:58.101 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.9.6, , ] 2021-09-06T12:50:58.101 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.100+0000 7fb0e8679380 0 bucket checkpoint complete 2021-09-06T12:50:58.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.117 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:58.168 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:58.169 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:58.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.191+0000 7f90e238e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:58.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.191+0000 7f90e238e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:58.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.191+0000 7f90e238e380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:58.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.191+0000 7f90e238e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:58.228 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.229+0000 7f90e238e380 1 bucket sync caught up with source: 2021-09-06T12:50:58.229 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.161.12, , ] 2021-09-06T12:50:58.229 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.161.12, , ] 2021-09-06T12:50:58.229 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.229+0000 7f90e238e380 0 bucket checkpoint complete 2021-09-06T12:50:58.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.247 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:58.297 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:58.297 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:58.320 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.321+0000 7f6d09a66380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:58.320 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.321+0000 7f6d09a66380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:58.321 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.321+0000 7f6d09a66380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:58.321 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.321+0000 7f6d09a66380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:58.359 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.360+0000 7f6d09a66380 1 bucket sync caught up with source: 2021-09-06T12:50:58.360 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.161.12, , ] 2021-09-06T12:50:58.360 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.161.12, , ] 2021-09-06T12:50:58.360 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.360+0000 7f6d09a66380 0 bucket checkpoint complete 2021-09-06T12:50:58.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.377 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:58.427 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:58.427 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:58.449 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.450+0000 7f9ec8de0380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:58.450 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.450+0000 7f9ec8de0380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:58.450 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.450+0000 7f9ec8de0380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:58.450 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.450+0000 7f9ec8de0380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:58.484 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.485+0000 7f9ec8de0380 1 bucket sync caught up with source: 2021-09-06T12:50:58.484 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.83.12, , ] 2021-09-06T12:50:58.485 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.83.12, , ] 2021-09-06T12:50:58.485 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:58.485+0000 7f9ec8de0380 0 bucket checkpoint complete 2021-09-06T12:50:58.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-24', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.501 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-24 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:58.559 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:58.559 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:58.585 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.584+0000 7f44362db380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:58.585 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.584+0000 7f44362db380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:58.585 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.584+0000 7f44362db380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:58.586 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.584+0000 7f44362db380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:58.620 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.619+0000 7f44362db380 1 bucket sync caught up with source: 2021-09-06T12:50:58.620 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.83.12, , ] 2021-09-06T12:50:58.621 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.83.12, , ] 2021-09-06T12:50:58.621 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:58.619+0000 7f44362db380 0 bucket checkpoint complete 2021-09-06T12:50:58.637 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-24 zones={test-zone2, test-zone1} 2021-09-06T12:50:58.643 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:58.687 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.687 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:58.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:58.741 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-24 zones={test-zone2, test-zone1} 2021-09-06T12:50:58.742 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-24 zones={test-zone3, test-zone1} 2021-09-06T12:50:58.748 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:58.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:58.799 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.845 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:58.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-24 zones={test-zone3, test-zone1} 2021-09-06T12:50:58.850 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-24 zones={test-zone3, test-zone2} 2021-09-06T12:50:58.856 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:58.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.903 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:58.907 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:58.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:58.958 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-24 zones={test-zone3, test-zone2} 2021-09-06T12:50:58.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:58.959 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.013 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.013 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.040 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.040+0000 7fb3e1ab2380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.040 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.040+0000 7fb3e1ab2380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.041 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.040+0000 7fb3e1ab2380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:59.041 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.040+0000 7fb3e1ab2380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.074 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.074+0000 7fb3e1ab2380 1 bucket sync caught up with source: 2021-09-06T12:50:59.075 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.075 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.075 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.074+0000 7fb3e1ab2380 0 bucket checkpoint complete 2021-09-06T12:50:59.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.092 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.146 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.146 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.171+0000 7fe32271b380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.171+0000 7fe32271b380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.171+0000 7fe32271b380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:59.171 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.171+0000 7fe32271b380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.214 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.215+0000 7fe32271b380 1 bucket sync caught up with source: 2021-09-06T12:50:59.215 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.215 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.215 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.215+0000 7fe32271b380 0 bucket checkpoint complete 2021-09-06T12:50:59.234 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.235 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.286 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.286 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.310 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.309+0000 7ffaf3f91380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.310 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.309+0000 7ffaf3f91380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.310 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.309+0000 7ffaf3f91380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:50:59.311 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.309+0000 7ffaf3f91380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.343 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.343+0000 7ffaf3f91380 1 bucket sync caught up with source: 2021-09-06T12:50:59.344 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.344 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.17.12, , ] 2021-09-06T12:50:59.344 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.343+0000 7ffaf3f91380 0 bucket checkpoint complete 2021-09-06T12:50:59.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.361 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.411 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.412 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.435 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.436+0000 7fee59403380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.436 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.436+0000 7fee59403380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.436 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.436+0000 7fee59403380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:59.436 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.436+0000 7fee59403380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.473 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.474+0000 7fee59403380 1 bucket sync caught up with source: 2021-09-06T12:50:59.474 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.534.6, , ] 2021-09-06T12:50:59.474 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.534.6, , ] 2021-09-06T12:50:59.474 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.474+0000 7fee59403380 0 bucket checkpoint complete 2021-09-06T12:50:59.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.491 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.542 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.542 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.565 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.566+0000 7f60249cb380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.566 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.566+0000 7f60249cb380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.566 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.566+0000 7f60249cb380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:50:59.566 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.566+0000 7f60249cb380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.604 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.604+0000 7f60249cb380 1 bucket sync caught up with source: 2021-09-06T12:50:59.604 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.534.6, , ] 2021-09-06T12:50:59.604 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.534.6, , ] 2021-09-06T12:50:59.604 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.604+0000 7f60249cb380 0 bucket checkpoint complete 2021-09-06T12:50:59.625 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.625 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.626 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.675 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.676 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.699+0000 7f9838a52380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.699+0000 7f9838a52380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.699+0000 7f9838a52380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:50:59.699 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.699+0000 7f9838a52380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.737 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.737+0000 7f9838a52380 1 bucket sync caught up with source: 2021-09-06T12:50:59.738 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.212.12, , ] 2021-09-06T12:50:59.738 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.212.12, , ] 2021-09-06T12:50:59.738 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:50:59.737+0000 7f9838a52380 0 bucket checkpoint complete 2021-09-06T12:50:59.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-25', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:50:59.755 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-25 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:50:59.809 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:50:59.809 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:50:59.833 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.833+0000 7fe51de3e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:50:59.834 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.833+0000 7fe51de3e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:50:59.834 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.833+0000 7fe51de3e380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:50:59.834 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.833+0000 7fe51de3e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:50:59.866 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.865+0000 7fe51de3e380 1 bucket sync caught up with source: 2021-09-06T12:50:59.866 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.212.12, , ] 2021-09-06T12:50:59.866 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.212.12, , ] 2021-09-06T12:50:59.867 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:50:59.865+0000 7fe51de3e380 0 bucket checkpoint complete 2021-09-06T12:50:59.883 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-25 zones={test-zone2, test-zone1} 2021-09-06T12:50:59.889 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:50:59.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:59.936 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:50:59.940 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:50:59.990 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:50:59.994 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-25 zones={test-zone2, test-zone1} 2021-09-06T12:50:59.995 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-25 zones={test-zone3, test-zone1} 2021-09-06T12:51:00.000 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:51:00.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:00.047 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:51:00.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:00.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:51:00.101 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-25 zones={test-zone3, test-zone1} 2021-09-06T12:51:00.102 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-25 zones={test-zone3, test-zone2} 2021-09-06T12:51:00.107 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:51:00.155 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:00.156 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:51:00.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:00.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:51:00.214 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-25 zones={test-zone3, test-zone2} 2021-09-06T12:51:00.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.215 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.216 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.267 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.290 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.290+0000 7f7b00012380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.291 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.290+0000 7f7b00012380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.291 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.290+0000 7f7b00012380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:51:00.291 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.290+0000 7f7b00012380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.324 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.323+0000 7f7b00012380 1 bucket sync caught up with source: 2021-09-06T12:51:00.324 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.324 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.324 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.323+0000 7f7b00012380 0 bucket checkpoint complete 2021-09-06T12:51:00.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.341 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.392 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.393 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.416 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.417+0000 7fd9b1d29380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.417 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.417+0000 7fd9b1d29380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.417 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.417+0000 7fd9b1d29380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:51:00.417 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.417+0000 7fd9b1d29380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.451 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.451+0000 7fd9b1d29380 1 bucket sync caught up with source: 2021-09-06T12:51:00.451 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.451 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.452 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.451+0000 7fd9b1d29380 0 bucket checkpoint complete 2021-09-06T12:51:00.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone1', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.474 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone1 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.524 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.525 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.549 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.549+0000 7f1760c0f380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.549 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.549+0000 7f1760c0f380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.549 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.549+0000 7f1760c0f380 1 Zone: test-zone4 (8c2685c1-df19-46c2-8a62-6cb4561f09e2) 2021-09-06T12:51:00.550 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.549+0000 7f1760c0f380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.581 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.581+0000 7f1760c0f380 1 bucket sync caught up with source: 2021-09-06T12:51:00.582 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.582 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000001.6.12, , ] 2021-09-06T12:51:00.582 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:00.581+0000 7f1760c0f380 0 bucket checkpoint complete 2021-09-06T12:51:00.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.598 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.598 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.649 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.650 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.674+0000 7f6a3e6e7380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.674+0000 7f6a3e6e7380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.674+0000 7f6a3e6e7380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:51:00.674 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.674+0000 7f6a3e6e7380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.723 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.723+0000 7f6a3e6e7380 1 bucket sync caught up with source: 2021-09-06T12:51:00.723 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.154.12, , ] 2021-09-06T12:51:00.723 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.154.12, , ] 2021-09-06T12:51:00.723 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.723+0000 7f6a3e6e7380 0 bucket checkpoint complete 2021-09-06T12:51:00.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone2', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.740 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone2 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.790 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.790 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.813 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.813+0000 7fef2155e380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.813 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.813+0000 7fef2155e380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.813 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.813+0000 7fef2155e380 1 Zone: test-zone3 (4561051f-a4da-4b2a-84e5-f27920586526) 2021-09-06T12:51:00.814 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.813+0000 7fef2155e380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.848 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.849+0000 7fef2155e380 1 bucket sync caught up with source: 2021-09-06T12:51:00.848 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000001.154.12, , ] 2021-09-06T12:51:00.849 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000001.154.12, , ] 2021-09-06T12:51:00.849 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.849+0000 7fef2155e380 0 bucket checkpoint complete 2021-09-06T12:51:00.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.865 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:00.915 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:00.915 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:00.938 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.939+0000 7f0515fc6380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:00.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.939+0000 7f0515fc6380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:00.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.939+0000 7f0515fc6380 1 Zone: test-zone1 (a488f9d2-4d4c-49b4-b839-c5d73c4d316e) 2021-09-06T12:51:00.939 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.939+0000 7f0515fc6380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:00.977 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.977+0000 7f0515fc6380 1 bucket sync caught up with source: 2021-09-06T12:51:00.977 INFO:teuthology.orchestra.run.smithi041.stderr: local status: [, , , , , , , , 00000000003.178.6, , ] 2021-09-06T12:51:00.977 INFO:teuthology.orchestra.run.smithi041.stderr: remote markers: [, , , , , , , , 00000000003.178.6, , ] 2021-09-06T12:51:00.977 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T12:51:00.977+0000 7f0515fc6380 0 bucket checkpoint complete 2021-09-06T12:51:00.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'checkpoint', '--bucket', 'hmrhlz-26', '--source-zone', 'test-zone3', '--retry-delay-ms', '5000', '--timeout-sec', '300', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '1', '--debug-ms', '0'] 2021-09-06T12:51:00.996 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync checkpoint --bucket hmrhlz-26 --source-zone test-zone3 --retry-delay-ms 5000 --timeout-sec 300 --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 1 --debug-ms 0 2021-09-06T12:51:01.050 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:01.051 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.074+0000 7f25e00c8380 1 Realm: test-realm (9b968e6d-c378-4ac6-abc0-926118720ae1) 2021-09-06T12:51:01.075 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.074+0000 7f25e00c8380 1 ZoneGroup: test-zonegroup (test-zonegroup) 2021-09-06T12:51:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.074+0000 7f25e00c8380 1 Zone: test-zone2 (b8f5da72-5870-48fa-9c1e-4ed10a947347) 2021-09-06T12:51:01.076 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.074+0000 7f25e00c8380 1 using period configuration: 588058fc-8215-429d-a24d-21295d48dfdc:4 2021-09-06T12:51:01.108 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.107+0000 7f25e00c8380 1 bucket sync caught up with source: 2021-09-06T12:51:01.108 INFO:teuthology.orchestra.run.smithi052.stderr: local status: [, , , , , , , , 00000000003.178.6, , ] 2021-09-06T12:51:01.108 INFO:teuthology.orchestra.run.smithi052.stderr: remote markers: [, , , , , , , , 00000000003.178.6, , ] 2021-09-06T12:51:01.109 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T12:51:01.107+0000 7f25e00c8380 0 bucket checkpoint complete 2021-09-06T12:51:01.125 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-26 zones={test-zone2, test-zone1} 2021-09-06T12:51:01.130 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:51:01.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.175 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:51:01.179 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:51:01.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-26 zones={test-zone2, test-zone1} 2021-09-06T12:51:01.230 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-26 zones={test-zone3, test-zone1} 2021-09-06T12:51:01.235 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:51:01.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.279 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:51:01.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:51:01.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-26 zones={test-zone3, test-zone1} 2021-09-06T12:51:01.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=hmrhlz-26 zones={test-zone3, test-zone2} 2021-09-06T12:51:01.339 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-09-06T12:51:01.383 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-09-06T12:51:01.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-09-06T12:51:01.434 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-09-06T12:51:01.438 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=hmrhlz-26 zones={test-zone3, test-zone2} 2021-09-06T12:51:01.439 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-09-06T12:51:01.439 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-27 2021-09-06T12:51:01.459 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-28 2021-09-06T12:51:01.496 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-29 2021-09-06T12:51:01.703 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-09-06T12:51:01.703 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-30 2021-09-06T12:51:01.723 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-31 2021-09-06T12:51:01.760 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-32 2021-09-06T12:51:01.979 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-09-06T12:51:01.980 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-33 2021-09-06T12:51:02.057 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-34 2021-09-06T12:51:02.233 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-35 2021-09-06T12:51:04.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-09-06T12:51:04.576 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-36 2021-09-06T12:51:04.603 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-37 2021-09-06T12:51:04.659 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-38 2021-09-06T12:51:04.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:04.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:04.706 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:04.751 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:04.751 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:04.803 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:51:04.824 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:51:04.824 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:04.824 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:51:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:51:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:51:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:51:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932664.685502_238.1", 2021-09-06T12:51:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:51:04.685502Z" 2021-09-06T12:51:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:04.828 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:51:04.829 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:51:04.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:04.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:04.830 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:04.874 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:04.875 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:05.131 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:05.132 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:05.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:05.134 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:05.135 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:05.136 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:05.137 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.138 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:05.139 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:05.140 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:05.141 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932663.430662_223.1", 2021-09-06T12:51:05.142 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:03.430662Z", 2021-09-06T12:51:05.143 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:05.144 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:05.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:05.145 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:05.146 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932663.430662_223.1 2021-09-06T12:51:10.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:10.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:10.149 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:10.209 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:10.209 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:10.372 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:10.377 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:10.378 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:10.379 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:10.380 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:10.381 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932663.430662_223.1", 2021-09-06T12:51:10.382 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:03.430662Z", 2021-09-06T12:51:10.383 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:10.384 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:10.385 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:10.385 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:10.386 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932663.430662_223.1 2021-09-06T12:51:15.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:15.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:15.389 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:15.496 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:15.497 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:16.034 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:16.035 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:16.035 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:16.035 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:16.035 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:16.036 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:16.036 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:16.036 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:16.036 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:16.036 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:16.037 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:16.037 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:16.037 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:16.038 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:16.038 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:16.039 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:16.039 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:16.039 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:16.039 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:16.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:16.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:16.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:16.040 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:16.041 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:16.041 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:16.041 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:16.041 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:16.041 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:16.042 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:16.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:16.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:16.042 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:16.043 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:16.043 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:16.043 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:16.043 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:16.043 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:16.044 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:16.044 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:16.044 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:16.044 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:16.044 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:16.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:16.045 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:16.045 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:16.046 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:16.046 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932663.430662_223.1", 2021-09-06T12:51:16.046 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:16.046 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:16.046 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:16.047 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:03.430662Z", 2021-09-06T12:51:16.047 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:16.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.047 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:16.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:16.048 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:16.048 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:16.048 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:16.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:16.049 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:16.050 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932663.430662_223.1 2021-09-06T12:51:21.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:21.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:21.052 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:21.141 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:21.141 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:21.452 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:21.452 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:21.453 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:21.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:21.455 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:21.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:21.457 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:21.458 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:21.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:21.460 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:21.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:21.462 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.594037_229.1", 2021-09-06T12:51:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:21.463 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.594037Z", 2021-09-06T12:51:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.464 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:21.465 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:21.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:21.466 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:21.467 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932664.594037_229.1 2021-09-06T12:51:26.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:26.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:26.470 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:26.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:26.517 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:26.581 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:26.581 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:26.582 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:26.582 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:26.582 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:26.582 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:26.583 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:26.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:26.583 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:26.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:26.584 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:26.584 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:26.584 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:26.584 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:26.585 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:26.585 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:26.585 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:26.585 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:26.585 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:26.586 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:26.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:26.586 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:26.587 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:26.587 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:26.587 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:26.587 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:26.587 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:26.588 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:26.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:26.588 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:26.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:26.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:26.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:26.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:26.589 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:26.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:26.590 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:26.590 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:26.590 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:26.591 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:26.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:26.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:26.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:26.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:26.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:26.592 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:26.592 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.588235_227.1", 2021-09-06T12:51:26.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:26.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:26.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:26.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.588235Z", 2021-09-06T12:51:26.593 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:26.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.594 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:26.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:26.595 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:26.595 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:26.595 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:26.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:26.595 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:26.597 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932664.588235_227.1 2021-09-06T12:51:28.370 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:51:28.370+0000 7f04fd82c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:51:31.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:31.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:31.600 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:31.679 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:31.680 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:31.797 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:31.798 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:31.798 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:31.798 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:31.799 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:31.799 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:31.799 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:31.799 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:31.800 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:31.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:31.800 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:31.800 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:31.800 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:31.801 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:31.801 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:31.801 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:31.801 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:31.801 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:31.802 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:31.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:31.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:31.803 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:31.803 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:31.803 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:31.803 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:31.803 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:31.804 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:31.804 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:31.804 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:31.804 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:31.804 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:31.805 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:31.805 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:31.805 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:31.806 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:31.806 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:31.806 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:31.806 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:31.806 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:31.807 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:31.807 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:31.807 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:31.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:31.808 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:31.808 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:31.808 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:31.808 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.588235_227.1", 2021-09-06T12:51:31.809 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:31.809 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:31.809 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:31.809 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.588235Z", 2021-09-06T12:51:31.809 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:31.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.810 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:31.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:31.810 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:31.811 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:31.811 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:31.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:31.811 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:31.812 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932664.588235_227.1 2021-09-06T12:51:36.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:36.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:36.815 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:36.871 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:36.871 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:37.312 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:37.313 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:37.313 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:37.313 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:37.313 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:37.314 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:37.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:37.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:37.314 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:37.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:37.315 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:37.315 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:37.315 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:37.315 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:37.316 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:37.316 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:37.316 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:37.316 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:37.317 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:37.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:37.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:37.317 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:37.318 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:37.318 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:37.318 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:37.318 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:37.318 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:37.319 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:37.319 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:37.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:37.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:37.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:37.320 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:37.320 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:37.320 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:37.320 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:37.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:37.322 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.673803_235.1", 2021-09-06T12:51:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:37.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.673803Z", 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:37.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:37.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:37.326 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:37.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932664.673803_235.1 2021-09-06T12:51:42.328 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:42.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:42.329 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:42.397 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:42.398 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:42.502 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:42.502 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:42.502 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:42.502 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:42.503 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:42.503 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:42.503 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:42.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:42.504 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:42.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:42.504 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:42.504 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:42.504 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:42.505 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:42.505 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:42.505 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:42.505 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:42.506 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:42.506 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:42.506 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:42.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:42.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:42.507 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:42.507 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:42.507 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:42.508 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:42.508 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:42.508 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:42.508 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:42.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:42.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:42.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:42.509 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:42.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:42.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:42.510 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:42.510 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:42.510 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:42.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:42.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:42.511 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:42.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.512 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:42.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:42.512 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:42.512 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:42.512 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:42.513 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.600083_230.1", 2021-09-06T12:51:42.513 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:42.513 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:42.513 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:42.514 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.600083Z", 2021-09-06T12:51:42.514 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:42.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.514 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:42.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:42.515 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:42.515 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:42.515 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:42.516 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:42.516 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:42.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932664.600083_230.1 2021-09-06T12:51:47.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:47.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:47.520 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:47.591 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:47.591 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:47.785 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:51:47.786 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:51:47.786 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:51:47.786 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:51:47.787 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:51:47.787 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:47.787 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:47.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:47.788 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:51:47.788 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:47.788 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:51:47.788 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:47.788 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:47.789 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:47.789 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:47.789 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:51:47.789 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:47.790 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:47.790 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:47.790 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.790 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:47.791 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:47.791 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:51:47.791 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:47.791 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:47.791 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:51:47.792 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:47.792 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:47.792 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:47.792 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:51:47.792 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:47.793 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.793 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:47.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:47.793 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:51:47.794 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:47.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:47.794 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:47.794 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:47.795 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:47.795 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:47.795 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:47.795 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:47.795 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:47.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:51:47.796 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:51:47.796 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:51:47.796 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:51:47.797 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932664.685502_238.1", 2021-09-06T12:51:47.797 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:51:47.797 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:51:47.797 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:51:47.797 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:04.685502Z", 2021-09-06T12:51:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:51:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:51:47.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:51:47.799 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:51:47.799 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:51:47.799 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:51:47.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:51:47.800 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:51:47.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:51:47.801 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:51:47.802 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:47.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:47.803 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:47.851 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:47.852 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:47.914 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:51:47.915 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:51:47.915 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:51:47.915 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:51:47.915 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:51:47.916 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:47.916 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:47.916 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:47.916 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:51:47.917 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:47.917 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:51:47.917 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:47.917 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:47.917 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:47.918 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:47.918 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:51:47.918 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:47.918 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:47.918 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:47.919 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.919 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:47.919 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:47.919 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:51:47.919 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:47.920 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:47.920 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:51:47.920 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:47.920 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:47.921 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:47.921 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:51:47.921 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:47.921 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.921 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:47.922 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:47.922 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:51:47.922 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:47.922 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:47.923 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:47.923 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:47.923 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:47.923 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:47.923 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:47.924 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:47.924 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.924 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:47.924 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:47.924 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:51:47.925 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:47.925 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:47.925 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:51:47.925 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:47.925 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:47.926 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:47.926 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:51:47.926 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:47.926 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.926 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.927 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:51:47.927 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:47.927 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:51:47.927 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:51:47.927 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:51:47.928 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:47.928 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:51:47.929 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:51:52.931 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:52.931 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:52.932 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:52.989 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:52.990 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:53.070 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:51:53.071 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:51:53.071 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:51:53.071 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:51:53.071 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:51:53.072 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:53.072 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:53.072 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:53.072 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:51:53.072 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:53.073 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:51:53.073 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:53.073 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:53.073 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:53.074 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:53.074 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:51:53.074 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:53.074 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:53.075 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:53.075 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.075 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:53.075 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:53.075 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:51:53.076 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:53.076 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:53.076 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:51:53.076 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:53.076 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:53.077 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:53.077 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:51:53.077 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:53.077 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.078 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:53.078 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:53.078 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:51:53.078 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:53.079 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:53.079 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:53.079 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:53.079 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:53.079 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:53.080 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:53.080 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:53.080 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.080 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:53.081 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:53.081 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:51:53.081 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:53.081 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:53.081 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:51:53.082 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:53.082 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:53.082 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:53.082 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:51:53.083 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:53.083 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.083 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.083 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:51:53.084 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:53.084 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:51:53.084 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:51:53.084 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:51:53.084 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:53.085 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:51:53.086 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:51:58.087 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:58.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:51:58.089 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:51:58.168 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:51:58.169 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:51:58.351 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:51:58.352 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:51:58.352 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:51:58.352 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:51:58.352 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:51:58.352 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:51:58.353 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:58.353 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:58.353 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:51:58.353 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:58.354 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:51:58.354 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:58.354 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:58.354 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:51:58.354 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:58.355 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:51:58.355 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:58.355 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:51:58.355 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:58.355 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.356 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:58.356 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:58.356 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:51:58.356 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:58.357 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:58.357 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:51:58.357 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:58.357 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:58.357 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:58.358 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:51:58.358 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:58.358 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.358 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:58.358 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:58.359 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:51:58.359 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:58.359 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:58.359 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:51:58.359 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:58.360 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:58.360 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:58.360 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:51:58.360 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:58.360 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.361 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:58.361 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:51:58.361 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:51:58.361 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:51:58.361 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:51:58.362 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932662.023602_208.1", 2021-09-06T12:51:58.362 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:51:58.362 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:51:58.362 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:51:58.362 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:02.023602Z", 2021-09-06T12:51:58.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:51:58.363 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.363 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.363 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:51:58.363 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:51:58.364 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:51:58.364 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:51:58.364 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:51:58.364 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:51:58.364 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:51:58.366 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932662.023602_208.1 2021-09-06T12:52:03.367 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:03.368 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:03.368 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:03.421 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:03.421 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:03.503 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:03.503 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:03.503 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:03.503 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:03.504 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:03.504 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:03.504 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:03.505 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:03.505 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:03.506 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:03.506 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:03.506 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:03.506 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:03.507 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:03.507 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:03.507 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:03.507 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:03.508 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:03.508 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:03.508 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.508 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:03.508 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:03.509 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:03.509 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:03.509 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:03.509 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:03.509 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:03.510 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:03.510 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:03.510 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:03.510 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:03.510 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.511 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:03.511 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:03.511 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:03.511 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:03.512 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:03.512 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:03.512 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:03.512 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:03.512 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:03.513 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:03.513 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:03.513 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.513 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:03.514 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:03.514 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:03.514 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:03.514 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:03.514 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T12:52:03.515 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:03.515 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:03.515 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:03.515 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T12:52:03.515 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:03.516 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.516 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.516 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:03.516 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:03.516 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:03.517 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:03.517 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:03.517 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:03.517 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:03.518 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.474951_171.1 2021-09-06T12:52:08.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:08.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:08.521 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:08.589 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:08.589 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:08.677 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:08.677 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:08.678 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:08.678 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:08.678 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:08.678 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:08.678 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:08.679 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:08.679 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:08.679 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:08.679 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:08.680 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:08.680 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:08.680 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:08.680 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:08.680 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:08.681 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:08.681 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:08.681 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:08.681 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.682 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:08.682 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:08.682 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:08.682 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:08.682 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:08.683 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:08.683 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:08.683 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:08.683 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:08.683 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:08.684 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:08.684 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.684 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:08.684 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:08.684 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:08.685 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:08.685 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:08.685 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:08.685 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:08.686 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:08.686 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:08.686 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:08.686 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:08.686 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.687 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:08.687 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:08.687 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:08.687 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:08.687 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T12:52:08.688 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:08.689 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.689 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.689 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:08.689 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:08.690 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:08.690 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:08.690 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:08.690 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:08.691 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:08.692 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.474951_171.1 2021-09-06T12:52:13.693 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:13.694 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:13.695 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:13.762 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:13.763 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:13.860 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:13.860 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:13.861 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:13.861 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:13.861 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:13.861 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:13.862 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:13.862 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:13.862 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:13.862 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:13.863 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:13.863 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:13.863 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:13.863 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:13.863 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:13.864 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:13.864 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:13.864 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:13.864 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:13.864 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.865 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:13.865 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:13.865 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:13.865 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:13.865 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:13.866 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:13.866 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:13.866 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:13.866 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:13.867 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:13.867 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:13.867 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.867 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:13.867 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:13.868 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:13.868 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:13.868 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:13.868 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:13.868 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:13.869 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:13.869 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:13.869 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:13.869 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:13.869 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.870 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:13.870 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:13.870 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:13.870 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:13.871 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:13.871 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:52:13.871 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:13.871 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:13.871 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:13.872 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:52:13.872 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:13.872 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.872 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.872 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:13.873 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:13.873 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:13.873 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:13.873 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:13.873 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:13.874 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:13.875 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:52:18.876 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:18.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:18.878 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:18.927 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:18.928 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:18.995 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:18.995 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:18.995 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:18.996 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:18.996 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:18.996 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:18.996 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:18.997 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:18.997 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:18.997 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:18.997 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:18.997 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:18.998 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:18.998 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:18.998 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:18.998 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:18.998 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:18.999 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:18.999 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:18.999 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:18.999 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:19.000 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:19.000 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:19.000 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:19.000 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:19.000 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:19.001 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:19.001 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:19.001 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:19.001 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:19.001 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:19.002 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:19.002 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:19.002 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:19.002 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:19.002 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:19.003 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:19.003 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:19.003 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:19.003 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:19.003 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:19.004 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:19.004 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:19.004 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:19.004 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:19.005 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:19.005 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:19.005 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:19.005 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:19.005 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:52:19.006 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:19.006 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:19.006 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:19.006 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:52:19.006 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:19.007 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:19.007 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:19.007 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:19.007 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:19.007 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:19.008 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:19.008 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:19.008 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:19.008 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:19.009 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:52:24.010 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:24.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:24.012 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:24.055 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:24.056 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:24.136 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:24.136 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:24.136 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:24.136 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:24.136 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:24.137 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:24.137 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:24.137 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:24.137 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:24.138 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:24.138 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:24.138 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:24.138 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:24.138 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:24.139 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:24.139 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:24.139 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:24.139 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:24.139 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:24.140 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.140 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:24.140 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:24.140 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:24.140 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:24.141 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:24.141 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:24.141 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:24.141 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:24.142 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:24.142 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:24.142 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:24.142 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.142 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:24.143 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:24.143 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:24.143 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:24.143 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:24.143 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:24.144 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:24.144 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:24.144 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:24.144 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:24.144 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:24.145 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.145 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:24.145 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:24.145 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:24.145 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:24.146 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:24.146 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:52:24.146 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:24.146 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:24.146 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:24.147 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:52:24.147 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:24.147 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.147 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.147 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:24.148 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:24.148 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:24.148 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:24.148 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:24.148 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:24.149 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:24.150 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:52:29.150 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:29.151 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:29.152 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:29.196 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:29.196 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:29.276 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:29.277 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:29.277 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:29.277 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:29.277 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:29.277 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:29.278 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:29.278 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:29.278 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:29.278 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:29.279 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:29.279 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:29.279 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:29.279 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:29.279 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:29.280 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:29.280 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:29.280 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:29.280 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:29.280 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.281 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:29.281 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:29.281 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:29.281 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:29.281 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:29.282 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:29.282 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:29.282 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:29.282 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:29.283 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:29.283 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:29.283 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.283 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:29.283 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:29.284 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:29.284 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:29.284 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:29.284 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:29.285 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:29.285 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:29.285 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:29.285 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:29.285 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:29.286 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.286 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:29.286 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:29.286 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:29.286 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:29.287 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:29.287 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:52:29.287 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:29.287 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:29.287 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:29.288 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:52:29.288 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:29.288 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.288 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.288 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:29.289 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:29.289 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:29.289 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:29.289 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:29.289 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:29.290 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:29.291 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:52:34.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:34.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:34.293 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:34.338 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:34.338 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:34.408 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:34.408 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:34.409 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:34.409 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:34.409 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:34.409 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:34.410 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:34.410 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:34.410 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:34.410 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:34.410 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:34.411 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:34.411 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:34.411 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:34.411 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:34.411 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:34.412 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:34.412 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:34.412 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:34.412 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.413 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:34.413 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:34.413 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:34.413 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:34.413 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:34.414 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:34.414 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:34.414 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:34.414 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:34.415 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:34.415 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:34.415 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.415 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:34.415 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:34.416 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:34.416 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:34.416 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:34.416 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:34.417 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:34.417 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:34.417 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:34.417 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:34.417 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:34.418 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.418 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:34.418 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:34.418 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:34.419 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:34.419 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:34.419 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:52:34.419 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:34.419 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:34.420 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:34.420 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:52:34.420 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:34.420 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.420 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.421 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:34.421 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:34.421 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:34.421 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:34.422 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:34.422 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:34.422 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:34.423 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:52:39.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:39.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:39.425 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:39.471 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:39.471 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:39.553 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:39.554 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:39.554 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:39.554 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:39.555 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:39.555 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:39.555 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:39.555 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:39.555 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:39.556 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:39.556 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:39.556 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:39.556 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:39.556 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:39.557 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:39.557 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:39.557 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:39.557 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:39.557 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:39.558 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.558 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:39.558 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:39.558 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:39.559 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:39.559 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:39.559 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:39.559 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:39.559 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:39.560 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:39.560 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:39.560 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:39.560 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.560 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:39.561 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:39.561 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:39.561 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:39.561 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:39.561 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:39.562 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:39.562 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:39.562 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:39.563 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:39.563 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:39.563 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.563 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:39.564 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:39.564 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:39.564 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:39.564 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:39.564 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.716832_185.1", 2021-09-06T12:52:39.565 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:39.565 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:39.565 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:39.565 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.716832Z", 2021-09-06T12:52:39.566 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:39.566 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.566 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.566 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:39.567 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:39.567 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:39.567 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:39.567 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:39.567 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:39.568 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:39.569 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.716832_185.1 2021-09-06T12:52:44.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:44.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:44.571 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:44.617 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:44.618 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:44.695 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:44.696 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:44.697 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:44.697 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:44.697 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:44.697 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:44.698 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:44.698 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:44.698 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:44.698 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:44.698 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:44.699 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:44.699 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:44.699 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:44.699 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:44.699 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.700 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:44.700 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:44.700 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:44.700 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:44.700 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:44.701 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:44.701 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:44.701 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:44.701 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:44.702 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:44.702 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:44.702 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.702 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:44.702 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:44.703 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:44.703 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:44.703 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:44.703 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:44.703 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:44.704 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:44.704 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:44.704 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:44.704 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:44.704 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.705 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:44.705 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:44.705 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:44.705 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:44.705 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:44.706 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:52:44.706 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:44.706 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:44.706 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:44.706 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:52:44.707 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:44.707 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.707 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.707 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:44.708 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:44.708 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:44.708 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:44.708 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:44.708 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:44.709 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:44.710 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:52:49.711 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:49.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:49.712 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:49.758 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:49.759 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:49.826 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:49.826 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:49.826 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:49.827 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:49.827 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:49.827 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:49.827 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:49.827 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:49.828 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:49.828 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:49.828 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:49.828 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:49.828 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:49.829 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:49.830 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:49.830 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.830 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:49.830 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:49.831 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:49.831 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:49.831 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:49.831 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:49.831 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:49.832 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:49.832 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:49.832 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:49.832 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:49.832 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.833 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:49.833 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:49.833 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:49.833 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:49.833 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:49.834 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:49.834 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:49.834 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:49.834 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:49.834 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:49.835 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:49.835 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.835 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:49.835 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:49.836 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:49.836 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:49.836 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:49.836 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:52:49.836 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:49.837 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:49.837 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:49.837 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:52:49.837 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:49.837 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.838 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.838 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:49.838 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:49.838 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:49.838 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:49.839 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:49.839 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:49.839 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:49.840 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:52:54.841 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:54.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:54.842 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:52:54.890 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:52:54.890 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:52:54.958 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:52:54.958 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:52:54.959 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:52:54.959 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:52:54.959 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:52:54.959 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:52:54.960 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:54.960 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:54.960 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:52:54.960 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:54.961 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:52:54.961 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:54.961 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:54.961 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:52:54.962 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:54.962 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:52:54.962 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:54.962 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:52:54.962 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:54.963 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.963 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:54.963 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:54.963 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:52:54.964 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:54.964 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:54.964 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:52:54.964 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:54.964 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:54.965 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:54.965 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:52:54.965 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:54.965 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.966 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:54.966 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:54.966 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:52:54.966 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:54.966 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:54.967 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:52:54.967 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:54.967 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:54.967 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:54.968 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:52:54.968 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:54.968 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.968 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:54.969 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:52:54.969 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:52:54.969 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:52:54.969 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:52:54.969 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:52:54.970 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:52:54.970 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:52:54.970 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:52:54.970 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:52:54.971 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:52:54.971 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.971 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.971 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:52:54.971 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:52:54.972 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:52:54.972 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:52:54.972 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:52:54.972 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:52:54.973 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:52:54.974 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:52:59.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:59.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:52:59.976 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:00.020 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:00.021 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:00.108 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:00.108 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:00.108 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:00.108 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:00.110 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:00.110 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:00.110 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:00.110 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:00.110 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:00.111 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:00.112 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.112 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:00.112 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:00.112 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:00.112 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:00.113 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:00.113 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:00.113 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:00.113 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:00.113 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:00.114 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:00.114 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:00.114 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.114 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:00.114 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:00.115 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:00.115 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:00.115 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:00.115 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:00.116 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:00.116 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:00.116 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:00.116 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:00.116 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:00.117 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.117 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:00.117 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:00.117 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:00.117 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:00.118 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:00.118 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:53:00.118 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:00.118 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:00.118 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:00.119 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:53:00.119 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:00.119 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.119 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.119 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:00.120 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:00.120 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:00.120 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:00.120 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:00.120 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:00.121 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:00.122 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:53:05.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:05.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:05.124 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:05.168 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:05.169 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:05.247 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:05.247 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:05.248 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:05.248 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:05.248 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:05.248 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:05.248 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:05.249 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:05.249 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:05.249 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:05.249 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:05.249 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:05.250 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:05.250 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:05.250 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:05.250 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:05.250 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:05.251 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:05.251 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:05.251 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.251 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:05.251 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:05.252 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:05.252 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:05.252 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:05.252 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:05.252 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:05.253 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:05.253 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:05.253 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:05.253 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:05.253 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.254 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:05.254 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:05.254 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:05.254 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:05.254 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:05.255 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:05.255 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:05.255 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:05.255 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:05.256 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:05.257 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:05.257 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:05.257 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:53:05.257 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:05.257 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:05.258 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:05.258 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:53:05.258 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:05.258 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.259 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.259 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:05.259 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:05.259 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:05.259 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:05.260 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:05.260 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:05.260 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:05.261 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.457681_169.1 2021-09-06T12:53:10.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:10.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:10.264 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:10.311 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:10.311 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:10.377 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:10.378 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:10.379 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:10.380 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:10.381 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:10.381 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:10.381 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:10.381 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:10.382 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.382 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:10.382 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:10.382 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:10.383 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:10.383 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:10.383 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:10.383 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:10.383 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:10.384 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:10.384 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:10.384 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:10.384 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.385 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:10.385 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:10.385 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:10.385 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:10.386 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:10.386 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:10.386 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:10.386 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:10.386 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:10.387 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:10.387 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:10.387 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.387 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:10.388 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:10.388 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:10.388 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:10.388 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:10.388 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:53:10.389 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:10.389 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:10.389 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:10.389 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:53:10.390 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:10.390 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.390 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.390 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:10.390 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:10.391 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:10.391 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:10.391 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:10.391 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:10.391 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:10.393 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:53:15.393 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:15.394 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:15.395 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:15.475 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:15.475 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:15.646 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:15.646 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:15.646 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:15.646 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:15.647 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:15.647 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:15.647 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:15.647 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:15.647 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:15.648 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:15.648 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:15.648 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:15.648 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:15.648 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:15.649 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:15.649 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:15.649 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:15.649 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:15.649 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:15.650 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.650 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:15.650 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:15.650 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:15.651 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:15.651 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:15.651 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:15.651 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:15.651 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:15.652 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:15.652 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:15.652 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:15.652 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.652 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:15.653 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:15.653 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:15.653 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:15.653 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:15.654 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:15.654 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:15.654 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:15.654 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:15.655 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:15.655 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:15.655 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.655 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:15.655 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:15.656 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:15.656 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:15.656 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:15.656 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:53:15.656 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:15.657 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:15.657 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:15.657 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:53:15.657 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:15.657 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.658 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.658 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:15.658 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:15.658 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:15.658 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:15.659 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:15.659 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:15.659 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:15.660 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.457681_169.1 2021-09-06T12:53:20.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:20.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:20.663 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:20.719 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:20.719 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:20.791 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:20.792 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:20.792 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:20.792 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:20.792 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:20.792 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:20.793 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:20.793 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:20.793 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:20.793 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:20.794 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:20.794 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:20.794 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:20.794 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:20.794 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:20.795 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:20.795 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:20.795 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:20.795 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:20.795 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.796 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:20.796 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:20.796 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:20.796 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:20.796 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:20.797 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:20.797 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:20.797 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:20.797 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:20.797 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:20.798 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:20.798 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.798 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:20.798 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:20.798 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:20.799 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:20.799 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:20.799 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:20.799 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:20.799 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:20.800 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:20.800 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:20.800 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:20.800 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.800 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:20.801 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:20.801 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:20.801 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:20.801 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:20.801 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:53:20.802 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:20.802 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:20.802 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:20.802 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:53:20.803 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:20.803 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.803 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.803 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:20.803 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:20.804 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:20.804 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:20.804 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:20.804 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:20.804 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:20.805 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.517790_176.1 2021-09-06T12:53:25.807 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:25.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:25.808 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:25.884 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:25.884 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:26.097 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:26.097 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:26.097 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:26.098 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:26.098 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:26.098 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:26.098 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:26.099 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:26.099 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:26.099 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:26.099 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:26.099 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:26.100 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:26.100 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:26.100 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:26.100 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:26.100 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:26.101 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:26.101 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:26.101 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.101 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:26.102 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:26.102 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:26.102 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:26.102 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:26.102 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:26.103 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:26.103 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:26.103 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:26.103 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:26.104 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:26.104 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.104 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:26.104 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:26.104 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:26.105 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:26.105 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:26.105 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:26.105 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:26.105 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:26.106 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:26.106 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:26.106 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:26.106 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.107 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:26.107 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:26.107 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:26.107 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:26.107 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:26.108 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:53:26.108 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:26.108 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:26.108 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:26.108 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:53:26.109 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:26.109 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.109 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.109 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:26.109 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:26.110 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:26.110 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:26.110 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:26.110 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:26.111 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:26.112 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:53:29.242 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T12:53:29.241+0000 7f89dff66700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:53:31.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:31.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:31.115 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:31.176 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:31.176 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:31.316 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:31.317 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:31.318 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:31.319 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:31.320 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:31.321 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:31.322 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:31.323 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:31.324 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:31.325 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:31.326 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.327 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:31.327 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:31.327 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:31.327 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:31.328 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:31.328 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:53:31.328 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:31.328 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:31.329 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:31.329 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:53:31.329 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:31.329 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.329 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.330 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:31.330 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:31.330 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:31.330 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:31.331 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:31.331 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:31.331 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:31.332 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:53:36.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:36.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:36.335 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:36.404 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:36.405 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:36.482 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:36.482 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:36.482 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:36.483 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:36.483 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:36.483 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:36.483 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:36.483 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:36.484 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:36.484 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:36.484 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:36.484 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:36.485 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:36.485 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:36.485 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:36.485 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:36.485 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:36.486 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:36.486 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:36.486 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.486 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:36.486 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:36.487 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:36.487 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:36.487 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:36.487 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:36.488 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:36.488 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:36.488 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:36.488 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:36.488 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:36.489 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.489 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:36.489 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:36.489 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:36.489 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:36.490 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:36.490 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:36.490 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:36.490 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:36.490 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:36.491 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:36.491 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:36.491 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.491 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:36.492 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:36.492 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:36.492 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:36.492 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:36.492 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T12:53:36.493 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:36.493 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:36.493 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:36.493 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T12:53:36.493 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:36.494 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.494 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.494 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:36.494 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:36.494 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:36.495 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:36.495 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:36.495 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:36.495 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:36.496 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.510450_174.1 2021-09-06T12:53:41.497 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:41.498 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:41.499 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:41.560 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:41.561 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:41.635 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:41.635 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:41.635 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:41.635 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:41.636 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:41.636 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:41.636 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:41.636 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:41.637 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:41.637 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:41.637 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:41.637 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:41.637 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:41.638 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:41.638 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:41.638 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:41.638 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:41.638 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:41.639 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:41.639 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.639 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:41.639 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:41.640 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:41.640 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:41.640 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:41.640 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:41.640 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:41.641 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:41.641 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:41.641 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:41.641 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:41.641 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.642 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:41.642 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:41.642 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:41.642 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:41.643 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:41.643 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:41.643 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:41.643 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:41.643 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:41.644 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:41.644 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:41.644 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.644 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:41.644 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:41.645 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:41.645 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:41.645 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:41.645 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:53:41.646 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:41.646 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:41.646 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:41.646 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:53:41.646 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:41.647 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.647 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.647 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:41.647 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:41.647 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:41.648 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:41.648 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:41.648 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:41.648 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:41.649 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:53:46.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:46.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:46.652 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:46.726 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:46.727 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:46.851 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:46.852 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:46.852 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:46.852 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:46.852 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:46.853 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:46.853 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:46.853 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:46.853 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:46.853 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:46.854 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:46.854 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:46.854 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:46.854 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:46.855 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:46.855 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:46.855 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:46.855 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:46.855 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:46.856 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.856 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:46.856 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:46.856 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:46.857 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:46.857 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:46.857 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:46.857 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:46.857 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:46.858 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:46.858 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:46.858 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:46.858 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.858 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:46.859 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:46.859 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:46.859 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:46.859 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:46.859 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:46.860 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:46.860 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:46.860 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:46.860 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:46.861 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:46.861 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.861 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:46.861 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:46.861 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:46.862 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:46.862 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:46.862 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:53:46.862 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:46.862 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:46.863 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:46.863 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:53:46.863 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:46.863 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.863 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.864 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:46.864 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:46.864 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:46.864 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:46.864 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:46.865 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:46.865 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:46.866 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:53:49.234 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:53:49.234+0000 7f04fc029700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:53:51.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:51.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:51.869 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:51.942 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:51.943 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:52.106 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:52.106 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:52.111 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:52.111 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:52.111 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:52.112 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:52.112 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:52.112 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:52.112 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:52.112 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:52.113 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:52.113 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:52.113 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:52.113 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:52.113 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:52.114 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:52.114 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:52.114 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:52.114 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:52.115 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.115 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:52.115 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:52.115 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:52.115 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:52.116 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:52.116 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:52.116 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:52.116 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:52.117 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:52.117 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:52.117 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:52.117 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.117 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:52.118 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:52.118 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:52.118 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:52.118 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:52.119 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:52.119 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:52.119 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:52.119 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:52.119 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:52.120 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:52.120 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.120 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:52.120 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:52.121 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:52.121 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:52.121 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:52.121 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:53:52.121 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:52.122 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:52.122 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:52.122 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:53:52.122 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:52.123 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.123 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.123 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:52.123 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:52.123 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:52.124 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:52.124 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:52.124 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:52.124 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:52.126 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:53:57.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:57.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:53:57.128 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:53:57.221 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:53:57.222 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:53:57.363 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:53:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:53:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:53:57.363 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:53:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:53:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:53:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:57.364 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:57.365 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:53:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:53:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:57.366 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:53:57.367 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:53:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:57.368 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:53:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.369 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:57.370 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:57.370 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:53:57.370 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:57.370 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:57.370 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:53:57.371 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:57.371 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:57.371 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:57.371 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:53:57.372 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:57.372 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.372 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:57.372 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:53:57.373 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:53:57.373 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:53:57.373 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:53:57.373 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:53:57.373 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:53:57.374 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:53:57.374 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:53:57.374 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:53:57.375 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:53:57.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.375 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:53:57.375 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:53:57.376 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:53:57.376 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:53:57.376 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:53:57.376 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:53:57.376 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:53:57.377 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.457681_169.1 2021-09-06T12:54:02.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:02.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:02.380 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:02.455 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:02.456 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:02.658 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:02.658 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:02.658 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:02.658 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:02.659 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:02.659 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:02.659 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:02.659 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:02.659 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:02.660 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:02.660 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:02.660 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:02.660 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:02.661 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:02.661 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:02.661 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:02.661 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:02.661 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:02.662 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:02.662 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.662 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:02.662 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:02.662 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:02.663 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:02.663 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:02.663 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:02.663 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:02.663 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:02.664 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:02.664 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:02.664 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:02.664 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.664 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:02.665 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:02.665 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:02.665 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:02.665 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:02.665 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:02.666 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:02.666 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:02.666 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:02.666 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:02.667 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:02.667 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.667 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:02.667 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:02.667 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:02.668 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:02.668 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:02.668 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T12:54:02.668 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:02.668 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:02.669 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:02.669 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T12:54:02.669 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:02.669 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.669 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.670 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:02.670 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:02.670 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:02.670 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:02.670 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:02.671 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:02.671 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:02.672 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.595266_180.1 2021-09-06T12:54:07.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:07.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:07.675 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:07.758 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:07.759 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:07.890 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:07.890 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:07.890 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:07.891 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:07.891 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:07.891 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:07.892 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:07.893 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:07.893 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:07.893 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:07.893 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:07.894 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:07.894 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:07.894 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:07.894 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:07.895 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.895 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:07.895 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:07.895 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:07.896 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:07.896 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:07.896 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:07.896 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:07.897 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:07.897 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:07.897 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:07.897 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:07.898 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.898 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:07.898 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:07.898 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:07.899 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:07.899 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:07.899 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:07.899 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:07.900 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:07.900 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:07.900 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:07.900 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:07.900 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.901 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:07.901 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:07.901 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:07.901 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:07.902 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:07.902 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T12:54:07.902 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:07.902 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:07.903 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:07.903 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T12:54:07.903 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:07.903 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.904 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.904 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:07.904 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:07.904 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:07.904 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:07.905 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:07.905 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:07.906 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.542715_178.1 2021-09-06T12:54:12.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:12.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:12.909 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:12.952 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:12.952 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:13.019 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:13.019 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:13.020 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:13.021 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:13.022 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:13.023 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:13.024 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:13.025 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:13.025 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:13.025 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:13.025 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:13.025 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.026 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:13.026 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:13.026 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:13.026 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:13.027 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:13.027 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:13.027 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:13.027 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:13.027 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:13.028 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:13.028 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:13.028 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.028 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:13.029 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:13.029 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:13.029 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:13.029 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:13.030 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:54:13.030 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:13.030 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:13.030 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:13.030 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:54:13.031 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:13.031 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.031 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.031 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:13.033 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:13.033 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:13.034 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:54:18.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:18.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:18.037 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:18.086 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:18.087 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:18.175 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:18.176 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:18.176 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:18.176 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:18.176 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:18.177 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:18.177 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:18.177 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:18.177 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:18.178 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:18.178 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:18.178 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:18.178 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:18.178 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:18.179 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:18.179 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:18.179 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:18.179 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:18.179 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:18.180 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.180 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:18.180 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:18.180 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:18.180 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:18.181 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:18.181 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:18.181 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:18.181 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:18.182 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:18.182 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:18.182 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:18.182 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.182 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:18.183 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:18.183 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:18.183 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:18.183 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:18.183 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:18.184 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:18.184 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:18.184 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:18.184 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:18.185 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:18.185 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.185 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:18.185 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:18.185 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:18.186 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:18.186 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:18.186 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:54:18.186 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:18.187 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:18.187 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:18.187 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:54:18.187 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:18.188 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.188 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.188 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:18.188 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:18.188 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:18.189 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:18.189 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:18.189 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:18.189 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:18.190 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:54:23.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:23.192 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:23.193 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:23.271 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:23.271 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:23.362 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:23.362 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:23.362 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:23.363 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:23.363 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:23.363 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:23.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:23.364 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:23.364 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:23.364 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:23.364 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:23.365 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:23.365 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:23.365 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:23.365 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:23.365 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:23.366 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:23.366 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:23.366 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:23.366 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.367 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:23.367 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:23.367 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:23.367 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:23.367 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:23.368 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:23.368 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:23.368 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:23.368 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:23.368 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:23.369 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:23.369 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.369 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:23.370 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:23.370 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:23.370 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:23.370 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:23.371 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:23.371 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:23.371 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:23.371 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:23.371 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:23.372 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:23.372 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.372 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:23.372 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:23.373 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:23.373 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:23.373 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:23.373 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:54:23.373 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:23.374 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:23.374 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:23.374 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:54:23.374 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:23.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.375 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:23.375 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:23.376 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:23.376 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:23.376 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:23.376 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:23.376 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:23.378 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.517790_176.1 2021-09-06T12:54:28.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:28.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:28.381 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:28.425 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:28.425 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:28.501 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:28.501 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:28.502 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:28.502 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:28.502 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:28.502 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:28.502 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:28.503 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:28.503 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:28.503 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:28.503 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:28.504 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:28.504 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:28.504 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:28.504 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:28.504 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:28.505 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:28.505 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:28.505 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:28.505 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.505 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:28.506 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:28.506 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:28.506 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:28.506 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:28.506 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:28.507 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:28.507 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:28.507 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:28.507 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:28.507 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:28.508 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.508 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:28.508 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:28.508 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:28.508 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:28.509 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:28.509 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:28.509 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:28.509 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:28.509 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:28.510 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:28.510 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:28.510 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.510 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:28.510 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:28.511 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:28.511 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:28.511 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:28.511 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:54:28.511 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:28.512 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:28.512 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:28.512 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:54:28.512 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:28.512 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.513 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.513 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:28.513 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:28.513 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:28.513 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:28.514 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:28.514 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:28.514 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:28.515 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.471383_170.1 2021-09-06T12:54:33.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:33.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:33.518 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:33.561 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:33.562 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:33.645 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:33.645 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:33.646 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:33.646 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:33.646 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:33.646 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:33.646 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:33.647 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:33.647 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:33.647 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:33.647 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:33.648 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:33.648 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:33.648 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:33.648 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:33.648 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:33.649 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:33.649 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:33.649 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:33.649 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.649 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:33.650 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:33.650 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:33.650 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:33.650 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:33.650 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:33.651 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:33.651 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:33.651 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:33.651 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:33.652 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:33.652 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.652 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:33.652 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:33.652 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:33.653 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:33.653 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:33.653 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:33.653 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:33.654 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:33.654 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:33.654 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:33.654 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:33.654 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.655 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:33.655 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:33.655 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:33.655 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:33.656 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:33.656 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:54:33.656 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:33.656 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:33.656 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:33.657 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:54:33.657 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:33.657 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.657 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.657 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:33.658 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:33.658 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:33.658 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:33.658 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:33.658 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:33.659 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:33.660 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:54:38.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:38.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:38.662 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:38.743 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:38.744 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:38.831 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:38.832 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:38.832 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:38.832 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:38.833 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:38.833 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:38.833 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:38.833 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:38.834 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:38.834 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:38.834 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:38.834 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:38.835 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:38.835 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:38.835 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:38.835 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:38.835 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:38.836 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:38.836 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:38.836 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.836 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:38.837 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:38.837 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:38.837 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:38.837 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:38.837 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:38.838 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:38.838 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:38.838 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:38.838 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:38.838 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:38.839 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.839 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:38.839 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:38.839 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:38.840 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:38.840 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:38.840 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:38.840 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:38.840 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:38.841 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:38.841 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:38.841 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:38.841 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.842 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:38.842 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:38.842 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:38.842 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:38.842 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:38.843 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:54:38.843 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:38.843 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:38.843 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:38.843 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:54:38.844 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:38.844 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.844 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.844 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:38.845 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:38.845 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:38.845 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:38.845 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:38.845 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:38.846 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:38.847 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:54:43.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:43.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:43.850 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:43.917 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:43.917 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:44.022 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:44.022 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:44.022 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:44.023 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:44.023 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:44.023 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:44.023 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:44.024 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:44.025 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:44.025 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:44.025 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:44.025 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:44.026 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:44.026 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:44.026 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:44.026 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.027 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:44.027 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:44.027 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:44.027 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:44.027 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:44.028 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:44.028 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:44.028 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:44.028 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:44.028 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:44.029 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:44.029 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.029 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:44.029 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:44.029 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:44.030 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:44.030 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:44.030 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:44.030 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:44.030 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:44.031 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:44.031 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:44.031 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:44.031 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.032 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:44.032 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:44.032 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:44.032 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:44.032 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:54:44.033 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:44.034 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.034 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.034 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:44.034 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:44.035 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:44.035 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:44.035 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:44.035 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:44.035 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:44.037 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:54:49.038 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:49.039 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:49.039 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:49.113 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:49.113 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:49.311 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:49.311 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:49.311 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:49.312 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:49.312 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:49.312 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:49.312 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:49.313 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:49.313 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:49.313 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:49.313 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:49.314 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:49.314 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:49.314 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:49.314 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:49.314 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:49.315 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:49.315 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:49.315 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:49.315 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.315 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:49.316 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:49.316 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:49.316 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:49.316 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:49.316 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:49.317 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:49.317 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:49.317 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:49.317 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:49.317 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:49.318 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.318 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:49.318 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:49.318 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:49.319 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:49.319 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:49.319 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:49.319 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:49.319 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:49.320 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:49.320 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:49.320 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:49.320 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.320 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:49.321 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:49.321 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:49.321 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:49.321 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:49.321 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:54:49.322 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:49.322 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:49.322 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:49.322 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:54:49.323 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:49.323 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.323 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.323 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:49.323 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:49.324 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:49.324 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:49.324 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:49.324 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:49.324 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:49.325 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:54:54.327 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:54.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:54.328 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:54.398 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:54.399 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:54.552 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:54.553 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:54.553 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:54.553 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:54.554 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:54.554 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:54.554 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:54.554 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:54.554 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:54.555 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:54.555 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:54.555 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:54.555 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:54.555 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:54.556 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:54.556 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:54.556 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:54.556 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:54.556 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:54.557 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.557 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:54.557 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:54.557 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:54.557 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:54.558 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:54.558 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:54.558 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:54.558 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:54.559 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:54.559 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:54.559 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:54.559 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.559 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:54.560 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:54.560 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:54.560 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:54.561 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:54.561 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:54.561 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:54.561 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:54.561 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:54.562 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:54.562 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:54.562 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.562 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:54.562 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:54.563 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:54.563 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:54.563 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:54.563 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:54:54.563 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:54.564 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:54.564 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:54.564 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:54:54.564 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:54.564 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.565 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.565 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:54.565 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:54.565 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:54.565 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:54.566 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:54.566 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:54.566 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:54.567 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.517790_176.1 2021-09-06T12:54:59.569 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:59.570 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:54:59.570 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:54:59.614 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:54:59.614 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:54:59.687 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:54:59.688 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:54:59.688 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:54:59.688 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:54:59.688 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:54:59.689 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:54:59.689 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:59.689 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:59.689 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:54:59.690 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:59.690 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:54:59.690 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:59.690 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:59.690 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:54:59.691 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:59.691 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:54:59.691 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:59.691 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:54:59.692 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:59.692 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.692 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:59.692 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:59.692 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:54:59.693 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:59.693 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:59.693 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:54:59.693 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:59.693 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:59.694 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:59.694 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:54:59.694 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:59.694 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.694 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:59.695 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:59.695 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:54:59.695 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:59.695 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:59.695 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:54:59.696 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:59.696 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:59.696 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:59.696 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:54:59.696 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:59.697 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.697 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:59.697 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:54:59.697 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:54:59.698 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:54:59.698 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:54:59.698 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T12:54:59.698 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:54:59.699 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:54:59.699 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:54:59.699 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T12:54:59.699 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:54:59.699 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.700 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.700 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:54:59.700 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:54:59.700 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:54:59.700 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:54:59.701 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:54:59.701 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:54:59.701 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:54:59.702 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.510450_174.1 2021-09-06T12:55:04.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:04.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:04.705 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:04.750 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:04.750 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:04.824 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:04.824 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:04.824 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:04.824 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:04.825 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:04.826 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:04.827 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:04.828 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:04.828 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:04.828 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:04.828 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.828 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:04.829 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:04.829 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:04.829 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:04.829 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:04.829 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:04.830 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:04.830 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:04.830 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:04.830 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:04.831 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:04.831 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.831 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:04.831 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:04.832 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:04.832 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:04.832 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:04.832 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:04.832 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:04.833 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:04.833 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:04.833 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:04.833 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:04.833 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.834 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:04.834 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:04.834 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:04.834 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:04.835 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:04.835 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:55:04.835 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:04.835 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:04.835 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:04.836 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:55:04.836 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:04.836 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.836 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.837 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:04.837 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:04.837 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:04.837 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:04.837 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:04.838 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:04.838 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:04.839 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.471383_170.1 2021-09-06T12:55:09.840 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:09.841 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:09.842 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:09.928 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:09.929 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:10.139 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:10.140 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:10.140 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:10.140 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:10.140 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:10.140 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:10.141 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:10.141 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:10.141 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:10.141 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:10.142 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:10.142 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:10.142 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:10.142 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:10.142 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:10.143 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:10.143 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:10.143 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:10.143 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:10.144 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.144 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:10.144 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:10.144 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:10.144 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:10.145 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:10.145 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:10.145 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:10.145 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:10.146 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:10.146 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:10.146 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:10.146 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.147 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:10.147 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:10.147 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:10.147 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:10.148 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:10.148 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:10.148 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:10.148 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:10.149 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:10.149 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:10.149 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:10.149 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.149 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:10.150 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:10.150 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:10.150 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:10.150 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:10.151 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:55:10.151 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:10.151 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:10.151 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:10.151 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:55:10.152 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:10.152 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.152 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.152 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:10.153 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:10.153 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:10.153 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:10.153 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:10.154 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:10.154 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:10.155 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:55:15.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:15.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:15.158 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:15.235 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:15.235 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:15.387 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:15.388 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:15.388 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:15.388 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:15.388 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:15.389 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:15.389 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:15.389 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:15.389 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:15.390 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:15.390 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:15.390 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:15.390 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:15.391 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:15.391 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:15.391 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:15.391 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:15.391 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:15.392 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:15.392 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.392 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:15.392 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:15.393 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:15.393 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:15.393 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:15.393 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:15.393 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:15.394 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:15.394 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:15.394 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:15.394 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:15.394 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.395 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:15.395 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:15.395 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:15.395 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:15.396 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:15.397 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:15.397 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:15.397 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.397 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:15.397 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:15.398 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:15.398 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:15.398 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:15.398 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:55:15.399 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:15.399 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:15.399 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:15.399 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:55:15.399 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:15.400 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.400 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.400 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:15.400 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:15.400 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:15.401 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:15.401 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:15.401 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:15.401 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:15.403 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:55:20.404 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:20.404 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:20.405 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:20.477 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:20.478 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:20.677 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:20.677 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:20.678 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:20.678 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:20.678 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:20.678 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:20.679 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:20.679 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:20.679 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:20.679 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:20.680 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:20.680 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:20.680 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:20.680 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:20.680 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:20.681 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:20.681 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:20.681 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:20.681 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:20.682 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.682 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:20.682 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:20.682 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:20.682 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:20.683 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:20.683 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:20.683 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:20.683 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:20.684 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:20.684 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:20.684 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:20.684 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.684 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:20.685 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:20.685 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:20.685 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:20.685 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:20.685 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:20.686 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:20.686 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:20.686 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:20.686 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:20.687 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:20.687 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.687 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:20.687 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:20.687 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:20.688 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:20.688 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:20.688 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:55:20.688 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:20.688 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:20.689 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:20.689 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:55:20.689 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:20.689 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.690 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.690 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:20.690 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:20.690 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:20.690 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:20.691 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:20.691 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:20.691 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:20.692 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.457681_169.1 2021-09-06T12:55:25.694 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:25.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:25.696 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:25.773 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:25.773 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:25.919 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:55:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:55:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:25.934 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:25.934 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:25.934 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:25.935 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:55:30.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:30.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:30.938 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:31.008 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:31.009 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:31.119 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:31.120 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:31.120 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:31.120 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:31.120 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:31.121 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:31.121 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:31.121 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:31.121 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:31.121 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:31.122 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:31.122 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:31.122 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:31.122 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:31.123 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:31.123 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:31.123 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:31.123 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:31.124 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:31.124 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.124 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:31.124 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:31.124 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:31.125 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:31.125 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:31.125 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:31.125 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:31.125 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:31.126 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:31.126 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:31.126 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:31.126 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.126 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:31.127 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:31.127 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:31.127 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:31.127 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:31.128 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:31.128 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:31.128 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:31.128 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:31.128 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:31.129 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:31.129 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.129 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:31.130 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:31.130 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:31.132 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:31.132 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:31.132 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:55:31.132 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:31.132 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:31.133 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:31.133 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:55:31.133 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:31.133 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.134 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.134 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:31.134 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:31.134 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:31.134 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:31.135 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:31.135 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:31.135 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:31.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:55:36.137 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:36.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:36.139 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:36.221 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:36.222 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:36.433 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:36.433 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:36.433 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:36.434 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:36.434 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:36.434 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:36.434 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:36.435 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:36.435 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:36.435 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:36.435 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:36.435 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:36.436 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:36.436 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:36.436 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:36.436 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:36.437 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:36.437 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:36.437 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:36.437 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.437 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:36.438 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:36.438 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:36.438 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:36.438 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:36.439 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:36.439 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:36.439 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:36.439 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:36.440 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:36.440 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:36.440 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.440 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:36.440 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:36.441 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:36.441 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:36.441 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:36.441 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:36.442 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:36.442 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:36.442 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:36.442 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:36.442 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:36.443 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.443 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:36.443 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:36.443 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:36.444 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:36.444 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:36.444 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:55:36.444 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:36.445 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:36.445 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:36.445 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:55:36.445 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:36.446 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.446 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.446 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:36.446 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:36.446 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:36.447 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:36.447 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:36.447 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:36.447 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:36.448 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.457681_169.1 2021-09-06T12:55:41.450 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:41.451 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:41.452 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:41.527 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:41.527 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:41.650 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:41.650 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:41.651 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:41.651 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:41.651 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:41.651 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:41.652 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:41.652 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:41.652 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:41.652 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:41.653 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:41.653 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:41.653 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:41.653 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:41.653 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:41.654 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:41.654 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:41.654 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:41.654 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:41.655 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.655 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:41.655 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:41.655 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:41.655 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:41.656 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:41.656 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:41.656 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:41.656 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:41.656 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:41.657 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:41.657 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:41.657 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.657 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:41.658 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:41.658 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:41.658 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:41.658 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:41.659 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:41.659 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:41.659 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:41.659 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:41.659 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:41.660 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:41.660 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.660 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:41.660 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:41.660 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:41.661 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:41.661 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:41.661 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:55:41.661 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:41.662 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:41.662 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:41.662 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:55:41.662 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:41.663 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.663 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.663 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:41.663 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:41.663 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:41.664 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:41.664 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:41.664 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:41.664 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:41.666 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:55:46.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:46.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:46.669 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:46.737 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:46.738 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:46.828 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:46.829 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:46.829 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:46.829 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:46.830 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:46.830 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:46.830 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:46.830 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:46.831 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:46.831 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:46.831 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:46.831 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:46.831 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:46.832 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:46.832 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:46.832 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:46.832 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:46.832 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:46.833 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:46.833 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.833 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:46.833 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:46.834 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:46.834 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:46.834 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:46.835 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:46.835 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:46.835 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:46.835 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:46.836 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:46.836 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:46.836 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.836 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:46.837 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:46.837 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:46.837 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:46.838 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:46.838 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:46.838 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:46.838 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:46.839 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:46.839 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:46.839 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:46.839 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.840 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:46.840 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:46.840 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:46.840 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:46.840 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:46.841 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:55:46.841 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:46.841 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:46.841 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:46.842 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:55:46.842 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:46.842 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.842 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.842 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:46.843 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:46.843 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:46.843 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:46.843 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:46.844 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:46.844 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:46.845 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:55:51.846 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:51.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:51.847 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:51.921 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:51.921 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:52.223 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:52.223 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:52.224 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:52.224 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:52.224 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:52.224 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:52.225 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:52.225 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:52.225 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:52.225 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:52.226 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:52.226 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:52.226 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:52.226 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:52.227 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:52.227 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:52.227 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:52.227 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:52.228 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:52.228 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.228 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:52.228 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:52.229 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:52.229 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:52.229 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:52.229 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:52.229 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:52.230 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:52.230 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:52.230 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:52.230 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:52.231 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.231 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:52.231 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:52.231 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:52.232 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:52.232 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:52.232 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:52.232 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:52.232 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:52.233 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:52.233 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:52.233 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:52.233 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.233 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:52.234 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:52.234 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:52.234 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:52.234 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:52.235 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:55:52.235 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:52.235 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:52.235 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:52.236 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:55:52.236 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:52.236 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.236 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.236 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:52.237 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:52.237 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:52.237 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:52.237 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:52.238 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:52.238 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:52.239 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:55:57.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:57.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:55:57.242 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:55:57.331 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:55:57.331 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:55:57.552 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:55:57.553 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:55:57.553 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:55:57.553 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:55:57.553 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:55:57.554 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:55:57.554 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:57.554 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:57.554 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:55:57.555 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:57.555 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:55:57.555 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:57.555 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:57.556 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:55:57.556 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:57.556 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:55:57.556 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:57.556 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:55:57.557 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:57.557 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.557 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:57.557 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:57.558 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:55:57.558 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:57.558 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:57.558 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:55:57.559 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:57.559 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:57.559 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:57.559 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:55:57.560 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:57.560 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.560 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:57.560 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:57.560 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:55:57.561 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:57.561 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:57.561 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:55:57.561 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:57.562 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:57.562 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:57.562 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:55:57.562 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:57.563 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.563 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:57.563 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:55:57.563 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:55:57.563 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:55:57.564 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:55:57.564 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:55:57.564 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:55:57.564 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:55:57.565 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:55:57.565 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:55:57.565 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:55:57.565 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.565 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.566 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:55:57.566 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:55:57.566 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:55:57.566 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:55:57.567 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:55:57.567 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:55:57.567 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:55:57.572 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.517790_176.1 2021-09-06T12:56:02.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:02.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:02.579 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:02.649 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:02.650 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:02.891 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:02.891 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:02.891 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:02.892 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:02.892 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:02.892 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:02.893 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:02.893 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:02.893 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:02.893 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:02.894 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:02.894 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:02.894 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:02.894 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:02.895 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:02.895 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:02.895 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:02.895 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:02.896 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:02.896 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.896 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:02.896 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:02.897 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:02.897 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:02.897 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:02.897 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:02.898 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:02.898 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:02.898 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:02.898 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:02.899 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:02.899 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.899 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:02.900 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:02.900 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:02.900 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:02.900 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:02.901 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:02.901 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:02.901 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:02.901 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:02.902 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:02.902 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:02.902 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.902 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:02.903 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:02.903 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:02.903 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:02.904 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:02.904 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.871397_203.1", 2021-09-06T12:56:02.904 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:02.904 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:02.905 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:02.905 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.871397Z", 2021-09-06T12:56:02.905 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:02.905 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.905 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.906 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:02.906 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:02.906 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:02.906 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:02.907 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:02.907 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:02.907 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:02.908 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.871397_203.1 2021-09-06T12:56:07.910 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:07.910 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:07.911 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:08.001 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:08.001 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:08.228 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:08.228 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:08.229 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:08.229 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:08.229 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:08.229 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:08.230 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:08.230 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:08.230 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:08.230 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:08.231 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:08.231 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:08.231 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:08.231 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:08.231 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:08.232 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:08.232 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:08.232 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:08.232 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:08.232 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.233 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:08.233 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:08.233 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:08.233 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:08.233 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:08.234 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:08.234 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:08.234 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:08.234 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:08.235 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:08.235 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:08.235 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.235 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:08.235 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:08.236 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:08.236 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:08.236 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:08.236 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:08.237 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:08.237 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:08.237 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:08.237 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:08.237 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:08.238 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.238 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:08.238 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:08.238 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:08.238 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:08.239 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:08.239 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:56:08.239 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:08.239 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:08.240 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:08.240 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:56:08.240 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:08.240 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.240 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.241 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:08.241 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:08.241 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:08.241 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:08.242 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:08.242 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:08.242 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:08.243 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:56:13.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:13.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:13.246 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:14.237 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:14.238 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:14.343 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:14.344 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:14.344 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:14.344 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:14.344 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:14.345 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:14.345 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:14.345 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:14.345 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:14.345 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:14.346 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:14.346 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:14.346 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:14.346 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:14.347 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:14.347 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:14.347 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:14.347 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:14.348 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:14.348 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.348 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:14.348 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:14.348 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:14.349 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:14.349 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:14.349 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:14.349 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:14.349 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:14.350 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:14.350 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:14.350 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:14.350 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.351 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:14.351 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:14.351 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:14.351 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:14.351 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:14.352 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:14.352 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:14.352 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:14.352 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:14.353 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:14.353 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:14.353 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.353 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:14.353 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:14.354 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:14.354 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:14.354 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:14.354 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:56:14.355 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:14.355 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:14.355 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:14.355 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:56:14.355 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:14.356 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.356 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.356 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:14.356 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:14.356 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:14.357 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:14.357 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:14.357 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:14.357 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:14.359 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:56:19.360 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:19.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:19.362 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:19.493 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:19.494 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:19.586 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:19.587 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:19.587 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:19.587 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:19.587 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:19.588 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:19.588 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:19.588 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:19.588 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:19.588 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:19.589 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:19.589 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:19.589 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:19.589 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:19.589 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:19.590 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:19.590 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:19.590 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:19.590 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:19.591 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.591 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:19.591 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:19.591 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:19.591 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:19.592 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:19.592 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:19.592 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:19.592 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:19.593 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:19.593 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:19.593 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:19.593 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.593 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:19.594 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:19.594 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:19.594 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:19.594 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:19.595 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:19.595 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:19.595 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:19.595 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:19.595 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:19.596 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:19.596 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.596 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:19.596 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:19.596 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:19.597 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:19.597 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:19.597 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:56:19.597 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:19.598 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:19.598 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:19.598 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:56:19.598 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:19.599 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.599 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.599 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:19.599 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:19.599 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:19.600 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:19.600 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:19.600 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:19.600 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:19.601 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.471383_170.1 2021-09-06T12:56:24.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:24.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:24.604 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:24.867 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:24.868 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:25.164 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:25.164 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:25.165 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:25.165 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:25.165 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:25.165 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:25.166 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:25.166 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:25.166 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:25.166 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:25.166 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:25.167 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:25.167 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:25.167 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:25.167 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:25.168 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:25.168 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:25.168 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:25.168 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:25.169 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.169 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:25.169 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:25.169 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:25.169 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:25.170 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:25.170 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:25.170 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:25.170 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:25.171 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:25.171 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:25.171 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:25.171 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.172 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:25.172 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:25.172 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:25.172 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:25.172 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:25.173 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:25.173 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:25.173 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:25.173 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:25.174 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:25.174 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:25.174 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.174 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:25.174 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:25.175 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:25.175 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:25.175 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:25.175 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:56:25.176 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:25.176 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:25.176 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:25.176 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:56:25.176 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:25.177 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.177 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.177 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:25.177 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:25.177 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:25.178 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:25.178 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:25.178 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:25.178 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:25.180 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.454323_168.1 2021-09-06T12:56:30.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:30.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:30.183 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:30.243 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:30.243 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:30.341 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:30.342 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:30.342 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:30.342 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:30.342 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:30.343 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:30.343 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:30.343 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:30.343 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:30.344 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:30.344 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:30.344 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:30.344 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:30.344 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:30.345 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:30.345 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:30.345 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:30.345 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:30.346 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:30.346 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.346 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:30.346 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:30.347 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:30.347 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:30.347 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:30.347 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:30.347 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:30.348 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:30.348 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:30.348 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:30.349 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:30.349 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.349 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:30.350 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:30.350 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:30.350 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:30.350 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:30.350 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:30.351 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:30.351 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:30.351 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:30.351 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:30.352 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:30.352 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.352 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:30.352 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:30.352 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:30.353 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:30.353 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:30.353 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:56:30.354 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:30.354 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:30.354 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:30.354 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:56:30.355 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:30.355 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.355 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.355 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:30.355 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:30.356 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:30.356 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:30.356 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:30.356 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:30.356 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:30.357 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.453043_167.1 2021-09-06T12:56:35.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:35.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:35.360 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:35.410 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:35.411 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:35.485 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:35.486 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:35.486 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:35.486 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:35.486 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:35.486 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:35.487 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:35.487 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:35.487 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:35.487 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:35.488 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:35.488 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:35.488 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:35.488 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:35.488 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:35.489 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:35.489 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:35.489 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:35.489 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:35.490 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.490 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:35.490 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:35.490 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:35.491 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:35.491 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:35.491 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:35.491 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:35.492 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:35.492 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:35.492 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:35.492 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:35.492 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.493 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:35.493 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:35.493 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:35.494 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:35.494 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:35.494 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:35.494 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:35.494 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:35.495 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:35.495 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:35.495 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:35.495 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.496 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:35.496 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:35.496 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:35.496 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:35.496 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:35.497 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:56:35.497 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:35.497 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:35.497 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:35.497 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:56:35.498 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:35.498 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.498 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.498 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:35.499 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:35.499 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:35.499 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:35.499 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:35.499 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:35.500 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:35.501 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.471383_170.1 2021-09-06T12:56:40.502 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:40.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:40.504 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:40.555 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:40.556 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:40.644 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:40.645 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:40.645 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:40.645 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:40.646 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:40.646 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:40.646 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:40.646 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:40.646 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:40.647 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:40.647 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:40.648 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:40.648 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:40.648 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:40.648 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:40.649 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:40.649 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:40.649 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:40.649 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:40.649 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.650 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:40.650 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:40.650 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:40.650 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:40.651 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:40.651 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:40.651 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:40.651 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:40.651 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:40.652 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:40.652 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:40.652 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.652 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:40.653 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:40.653 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:40.653 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:40.654 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:40.654 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:40.654 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:40.654 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:40.655 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:40.655 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:40.655 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:40.655 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.655 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:40.656 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:40.656 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:40.656 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:40.656 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:40.657 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:56:40.657 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:40.657 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:40.657 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:40.657 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:56:40.658 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:40.658 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.659 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.659 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:40.659 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:40.659 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:40.659 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:40.660 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:40.660 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:40.660 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:40.661 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.476870_172.1 2021-09-06T12:56:45.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:45.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:45.664 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:45.714 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:45.714 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:45.787 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:45.787 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:45.787 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:45.788 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:45.788 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:45.788 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:45.788 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:45.789 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:45.789 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:45.789 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:45.789 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:45.789 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:45.790 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:45.790 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:45.790 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:45.790 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:45.791 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:45.791 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:45.791 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:45.791 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.792 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:45.792 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:45.792 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:45.792 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:45.792 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:45.793 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:45.793 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:45.793 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:45.793 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:45.793 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:45.794 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:45.794 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.794 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:45.794 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:45.795 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:45.795 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:45.795 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:45.795 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:45.795 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:45.796 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:45.796 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:45.796 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:45.796 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:45.797 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.797 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:45.797 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:45.797 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:45.797 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:45.798 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:45.798 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T12:56:45.798 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:45.798 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:45.798 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:45.799 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T12:56:45.799 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:45.799 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.799 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.799 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:45.800 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:45.800 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:45.800 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:45.800 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:45.801 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:45.801 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:45.802 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.713479_184.1 2021-09-06T12:56:50.803 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:50.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:50.805 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:50.851 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:50.851 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:50.940 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:50.940 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:50.941 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:50.942 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:50.943 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:50.944 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:50.945 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:50.946 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.947 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:50.948 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:50.949 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:50.950 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:50.950 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:50.950 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:50.950 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.951 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:50.951 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:50.951 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:50.951 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:50.952 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:50.952 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T12:56:50.952 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:50.952 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:50.953 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:50.953 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T12:56:50.953 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:50.953 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.953 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.954 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:50.954 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:50.954 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:50.954 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:50.955 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:50.955 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:50.955 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:50.956 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.713479_184.1 2021-09-06T12:56:55.957 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:55.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:56:55.959 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:56:56.011 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:56:56.012 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:56:56.090 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:56:56.090 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:56:56.091 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:56:56.091 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:56:56.091 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:56:56.091 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:56:56.092 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:56.092 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:56.092 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:56:56.092 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:56.093 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:56:56.093 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:56.093 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:56.093 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:56:56.093 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:56.094 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:56:56.094 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:56.094 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:56:56.094 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:56.095 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.095 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:56.095 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:56.095 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:56:56.095 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:56.096 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:56.096 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:56:56.096 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:56.096 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:56.097 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:56.097 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:56:56.097 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:56.097 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.098 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:56.098 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:56.098 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:56:56.098 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:56.098 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:56.099 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:56:56.099 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:56.099 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:56.099 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:56.100 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:56:56.100 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:56.100 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.100 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:56.101 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:56:56.101 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:56:56.101 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:56:56.101 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:56:56.102 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:56:56.102 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:56:56.102 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:56:56.102 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:56:56.103 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:56:56.103 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:56:56.103 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.103 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.103 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:56:56.104 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:56:56.104 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:56:56.104 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:56:56.104 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:56:56.105 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:56:56.105 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:56:56.106 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630932664.685502_238.1 target=1_1630932661.471383_170.1 2021-09-06T12:57:01.109 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... FAIL 2021-09-06T12:57:01.111 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-39 2021-09-06T12:57:01.143 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-40 2021-09-06T12:57:01.236 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-41 2021-09-06T12:57:01.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:01.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:01.285 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:01.335 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:01.336 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:01.395 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T12:57:01.413 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T12:57:01.414 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:01.414 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:01.414 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T12:57:01.415 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:01.415 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:01.415 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:01.415 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T12:57:01.416 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:01.416 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:01.416 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:01.416 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T12:57:01.416 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:01.417 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:01.417 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630933021.264676_258.1", 2021-09-06T12:57:01.417 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:57:01.264676Z" 2021-09-06T12:57:01.417 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:01.418 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T12:57:01.419 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T12:57:01.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:01.420 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:01.421 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:01.530 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:01.530 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:01.597 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:57:01.598 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:57:01.598 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:57:01.598 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:57:01.598 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:57:01.599 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:01.599 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:01.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:01.599 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:57:01.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:01.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:57:01.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:01.600 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:01.600 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:01.601 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:01.601 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:57:01.601 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:01.601 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:01.602 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:01.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:01.602 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:01.602 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:57:01.603 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:01.603 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:01.603 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:57:01.603 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:01.604 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:01.604 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:01.604 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:57:01.604 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:01.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:01.605 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:01.605 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:57:01.605 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:01.606 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:01.607 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:01.607 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:01.607 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:57:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:01.608 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933021.129094_247.1", 2021-09-06T12:57:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:01.609 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:57:01.129094Z", 2021-09-06T12:57:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:57:01.610 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:01.614 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:57:01.614 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:57:01.614 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:57:01.615 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:01.615 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:57:01.616 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630933021.129094_247.1 2021-09-06T12:57:06.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:06.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:06.619 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:06.670 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:06.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:06.744 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T12:57:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T12:57:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T12:57:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T12:57:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T12:57:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T12:57:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T12:57:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T12:57:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:06.749 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T12:57:06.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:06.749 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:06.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T12:57:06.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:06.750 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:06.750 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:06.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T12:57:06.750 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:06.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:06.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:06.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T12:57:06.751 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:06.752 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:06.752 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:06.752 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:06.752 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:06.753 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:06.753 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:06.753 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:06.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:06.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T12:57:06.754 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T12:57:06.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T12:57:06.754 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T12:57:06.754 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933021.264676_258.1", 2021-09-06T12:57:06.755 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T12:57:06.755 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T12:57:06.755 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T12:57:06.755 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:57:01.264676Z", 2021-09-06T12:57:06.756 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T12:57:06.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.756 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T12:57:06.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T12:57:06.757 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T12:57:06.757 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T12:57:06.757 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T12:57:06.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T12:57:06.758 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T12:57:06.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T12:57:06.759 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T12:57:06.760 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:06.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:06.761 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:06.814 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:06.815 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:06.911 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:06.912 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:06.912 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:06.912 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:06.912 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:06.913 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:06.913 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:06.913 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:06.913 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:06.914 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:06.914 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:06.914 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:06.914 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:06.914 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:06.915 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:06.915 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:06.915 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:06.915 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:06.916 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:06.916 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.916 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:06.916 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:06.916 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:06.917 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:06.917 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:06.917 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:06.917 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:06.918 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:06.918 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:06.918 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:06.918 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:06.918 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.919 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:06.919 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:06.919 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:06.919 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:06.920 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:06.920 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:06.920 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:06.920 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:06.920 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:06.921 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:06.921 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:06.921 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.921 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:06.922 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:06.922 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:06.922 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:06.922 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:06.922 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T12:57:06.923 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:06.923 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:06.923 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:06.923 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T12:57:06.924 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:06.924 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.924 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.924 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:06.924 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:06.925 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:06.925 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:06.925 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:06.925 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:06.925 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:06.927 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.542715_178.1 2021-09-06T12:57:11.928 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:11.946 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:11.947 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:11.995 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:11.996 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:12.067 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:12.067 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:12.068 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:12.068 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:12.068 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:12.068 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:12.069 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:12.069 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:12.069 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:12.069 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:12.070 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:12.070 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:12.070 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:12.070 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:12.071 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:12.071 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:12.071 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:12.071 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:12.072 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:12.072 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.072 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:12.072 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:12.073 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:12.073 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:12.073 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:12.073 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:12.073 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:12.074 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:12.074 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:12.074 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:12.074 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:12.075 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.075 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:12.075 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:12.075 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:12.075 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:12.076 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:12.076 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:12.076 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:12.076 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:12.077 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:12.077 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:12.077 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:12.077 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.077 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:12.078 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:12.078 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:12.078 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:12.078 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:12.079 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:57:12.079 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:12.079 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:12.079 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:12.079 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:57:12.080 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:12.080 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.080 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.080 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:12.081 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:12.081 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:12.081 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:12.081 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:12.081 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:12.082 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:12.083 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T12:57:17.084 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:17.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:17.085 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:17.250 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:17.251 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:17.326 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:17.326 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:17.326 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:17.327 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:17.327 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:17.327 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:17.327 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:17.328 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:17.328 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:17.328 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:17.328 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:17.328 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:17.329 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:17.329 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:17.329 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:17.329 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:17.329 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:17.330 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:17.330 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:17.330 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.330 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:17.331 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:17.331 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:17.331 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:17.331 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:17.331 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:17.332 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:17.332 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:17.332 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:17.332 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:17.332 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:17.333 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.333 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:17.333 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:17.333 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:17.334 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:17.334 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:17.334 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:17.334 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:17.334 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:17.335 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:17.335 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:17.335 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:17.335 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.336 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:17.336 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:17.336 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:17.336 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:17.336 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:17.337 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T12:57:17.337 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:17.337 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:17.337 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:17.338 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T12:57:17.338 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:17.338 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.338 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.338 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:17.339 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:17.339 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:17.339 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:17.339 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:17.339 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:17.340 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:17.341 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.595266_180.1 2021-09-06T12:57:22.343 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:22.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:22.344 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:22.404 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:22.404 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:22.481 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:22.482 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:22.482 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:22.482 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:22.482 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:22.483 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:22.483 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:22.483 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:22.483 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:22.484 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:22.484 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:22.484 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:22.484 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:22.484 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:22.485 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:22.485 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:22.485 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:22.485 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:22.485 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:22.486 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.486 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:22.486 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:22.486 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:22.487 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:22.487 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:22.487 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:22.487 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:22.487 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:22.488 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:22.488 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:22.488 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:22.488 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.488 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:22.489 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:22.489 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:22.489 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:22.489 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:22.490 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:22.490 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:22.490 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:22.490 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:22.491 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:22.491 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:22.491 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.491 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:22.491 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:22.492 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:22.492 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:22.492 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:22.492 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T12:57:22.493 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:22.493 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:22.493 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:22.493 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T12:57:22.493 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:22.494 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.494 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.494 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:22.494 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:22.495 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:22.495 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:22.495 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:22.495 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:22.495 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:22.497 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.718130_186.1 2021-09-06T12:57:27.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:27.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:27.499 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:27.723 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:27.723 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:27.795 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:27.796 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:27.796 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:27.797 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:27.797 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:27.797 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:27.797 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:27.798 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:27.798 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:27.798 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:27.798 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:27.799 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:27.799 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:27.799 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:27.799 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:27.799 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:27.800 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:27.800 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:27.800 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:27.800 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.800 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:27.801 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:27.801 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:27.801 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:27.801 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:27.802 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:27.802 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:27.802 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:27.802 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:27.802 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:27.803 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:27.804 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:27.804 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:27.804 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:27.804 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:27.805 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:27.805 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:27.805 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:27.805 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:27.805 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.806 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:27.806 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:27.806 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:27.806 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:27.807 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:27.807 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:57:27.807 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:27.807 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:27.808 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:27.808 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:57:27.808 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:27.808 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.808 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.809 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:27.809 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:27.809 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:27.809 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:27.810 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:27.810 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:27.810 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:27.811 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T12:57:32.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:32.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:32.814 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:32.888 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:32.888 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:33.212 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:33.212 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:33.213 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:33.213 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:33.214 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:33.215 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:33.215 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:33.216 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:33.216 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:33.216 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:33.217 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:33.217 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:33.218 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:33.218 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:33.218 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:33.219 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:33.219 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:33.219 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:33.220 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:33.220 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.221 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:33.221 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:33.221 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:33.222 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:33.222 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:33.222 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:33.223 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:33.223 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:33.224 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:33.224 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:33.224 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:33.225 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.225 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:33.225 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:33.226 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:33.226 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:33.227 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:33.227 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:33.228 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:33.229 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:33.229 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:33.230 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:33.230 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:33.230 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.231 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:33.231 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:33.231 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:33.231 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:33.232 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:33.232 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T12:57:33.232 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:33.233 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:33.233 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:33.233 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T12:57:33.234 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:33.234 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.235 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.235 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:33.235 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:33.236 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:33.236 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:33.238 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:33.238 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:33.238 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:33.240 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.516029_175.1 2021-09-06T12:57:38.242 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:38.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:38.243 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:38.341 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:38.341 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:38.503 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:38.512 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:38.512 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:38.512 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:38.512 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:38.513 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.513 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:38.513 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:38.513 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:38.513 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:38.514 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:38.514 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:57:38.514 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:38.514 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:38.515 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:38.515 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:57:38.515 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:38.515 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.515 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.516 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:38.516 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:38.516 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:38.516 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:38.516 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:38.517 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:38.517 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:38.518 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T12:57:43.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:43.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:43.521 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:43.571 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:43.571 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:43.762 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:43.762 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:43.762 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:43.763 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:43.763 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:43.763 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:43.763 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:43.763 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:43.764 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:43.764 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:43.764 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:43.765 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:43.765 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:43.765 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:43.765 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:43.765 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:43.766 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:43.766 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:43.766 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:43.766 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.767 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:43.767 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:43.767 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:43.767 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:43.768 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:43.768 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:43.768 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:43.768 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:43.768 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:43.769 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:43.769 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:43.769 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.769 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:43.769 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:43.770 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:43.770 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:43.770 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:43.770 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:43.771 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:43.771 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:43.771 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:43.771 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:43.771 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:43.772 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.772 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:43.772 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:43.772 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:43.773 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:43.773 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:43.773 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T12:57:43.773 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:43.773 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:43.774 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:43.774 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T12:57:43.774 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:43.774 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.775 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.775 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:43.775 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:43.775 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:43.775 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:43.776 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:43.776 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:43.776 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:43.776 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.595266_180.1 2021-09-06T12:57:48.778 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:48.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:48.780 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:48.866 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:48.866 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:49.019 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:49.019 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:49.019 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:49.020 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:49.020 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:49.020 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:49.020 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:49.021 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:49.021 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:49.021 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:49.021 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:49.021 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:49.022 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:49.022 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:49.022 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:49.022 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:49.023 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:49.023 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:49.023 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:49.023 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.024 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:49.024 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:49.024 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:49.024 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:49.024 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:49.025 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:49.025 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:49.025 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:49.025 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:49.026 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:49.026 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:49.026 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.026 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:49.026 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:49.027 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:49.027 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:49.027 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:49.027 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:49.028 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:49.028 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:49.028 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:49.028 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:49.028 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:49.029 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.029 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:49.029 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:49.029 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:49.030 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:49.030 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:49.030 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T12:57:49.030 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:49.030 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:49.031 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:49.031 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T12:57:49.031 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:49.031 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.031 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.032 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:49.032 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:49.032 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:49.032 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:49.033 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:49.033 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:49.033 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:49.034 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T12:57:54.036 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:54.037 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:54.037 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:54.132 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:54.132 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:57:54.612 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:57:54.613 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:57:54.613 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:57:54.613 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:57:54.613 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:57:54.613 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:57:54.614 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:54.614 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:54.614 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:57:54.614 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:54.615 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:57:54.615 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:54.615 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:54.615 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:57:54.616 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:54.616 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:57:54.616 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:54.616 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:57:54.616 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:54.617 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.617 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:54.617 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:54.617 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:57:54.618 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:54.618 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:54.618 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:57:54.618 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:54.618 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:54.619 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:54.619 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:57:54.619 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:54.619 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.620 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:54.620 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:54.620 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:57:54.620 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:54.620 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:54.621 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:57:54.621 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:54.621 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:54.621 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:54.622 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:57:54.622 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:54.622 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.623 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:54.623 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:57:54.623 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:57:54.623 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:57:54.624 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:57:54.624 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T12:57:54.624 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:57:54.624 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:57:54.625 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:57:54.625 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T12:57:54.625 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:57:54.625 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.625 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.626 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:57:54.626 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:57:54.626 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:57:54.626 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:57:54.627 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:57:54.627 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:57:54.627 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:57:54.628 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T12:57:59.630 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:59.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:57:59.632 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:57:59.713 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:57:59.714 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:00.094 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:00.095 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:00.095 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:00.095 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:00.096 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:00.096 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:00.096 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:00.096 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:00.096 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:00.097 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:00.097 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:00.097 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:00.097 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:00.098 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:00.098 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:00.098 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:00.098 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:00.099 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:00.099 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:00.099 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.099 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:00.099 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:00.100 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:00.100 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:00.100 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:00.100 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:00.101 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:00.101 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:00.101 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:00.101 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:00.101 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:00.102 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.102 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:00.102 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:00.102 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:00.103 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:00.103 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:00.103 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:00.103 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:00.104 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:00.104 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:00.104 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:00.105 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:00.105 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.105 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:00.105 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:00.105 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:00.106 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:00.106 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:00.106 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:00.106 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:00.107 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:00.107 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:00.107 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:00.107 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:00.107 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.108 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.108 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:00.109 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:00.110 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:00.111 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:05.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:05.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:05.114 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:05.211 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:05.211 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:05.782 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:05.782 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:05.782 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:05.783 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:05.783 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:05.783 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:05.783 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:05.784 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:05.784 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:05.785 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:05.785 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:05.785 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:05.785 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:05.785 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:05.786 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:05.786 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:05.786 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:05.786 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:05.787 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:05.787 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.787 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:05.787 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:05.787 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:05.788 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:05.788 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:05.788 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:05.788 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:05.789 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:05.789 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:05.789 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:05.789 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:05.789 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.790 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:05.790 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:05.790 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:05.790 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:05.791 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:05.791 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:05.791 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:05.791 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:05.792 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:05.792 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:05.792 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:05.792 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.793 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:05.793 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:05.793 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:05.793 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:05.793 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:05.794 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:05.794 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:05.794 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:05.794 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:05.795 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:05.795 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:05.795 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.795 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.795 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:05.796 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:05.796 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:05.796 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:05.796 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:05.797 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:05.797 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:05.798 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:10.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:10.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:10.801 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:10.871 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:10.872 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:11.434 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:11.434 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:11.434 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:11.434 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:11.435 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:11.435 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:11.435 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:11.435 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:11.436 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:11.436 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:11.436 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:11.436 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:11.436 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:11.437 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:11.437 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:11.437 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:11.437 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:11.438 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:11.438 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:11.438 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.438 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:11.438 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:11.439 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:11.439 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:11.439 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:11.439 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:11.440 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:11.440 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:11.440 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:11.440 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:11.440 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:11.441 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.441 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:11.441 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:11.441 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:11.442 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:11.442 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:11.442 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:11.442 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:11.442 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:11.443 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:11.443 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:11.443 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:11.443 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.444 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:11.444 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:11.444 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:11.444 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:11.445 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:11.445 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:11.445 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:11.445 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:11.445 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:11.446 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:11.446 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:11.446 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.446 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.447 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:11.447 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:11.447 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:11.447 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:11.447 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:11.448 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:11.448 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:11.448 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:16.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:16.450 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:16.451 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:16.557 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:16.557 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:17.006 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:17.006 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:17.006 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:17.007 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:17.007 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:17.007 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:17.007 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:17.008 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:17.008 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:17.008 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:17.008 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:17.009 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:17.009 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:17.009 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:17.009 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:17.010 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:17.010 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:17.010 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:17.010 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:17.010 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.011 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:17.011 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:17.011 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:17.011 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:17.012 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:17.012 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:17.012 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:17.012 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:17.013 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:17.013 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:17.013 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:17.013 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.013 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:17.014 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:17.014 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:17.014 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:17.014 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:17.014 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:17.015 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:17.015 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:17.015 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:17.015 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:17.016 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:17.016 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.016 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:17.016 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:17.016 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:17.017 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:17.017 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:17.017 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:17.017 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:17.018 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:17.018 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:17.018 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:17.018 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:17.019 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.019 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.019 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:17.019 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:17.020 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:17.020 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:17.020 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:17.020 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:17.021 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:17.022 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:22.023 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:22.024 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:22.024 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:22.111 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:22.112 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:22.527 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:22.528 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:22.528 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:22.528 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:22.528 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:22.528 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:22.529 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:22.529 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:22.529 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:22.529 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:22.530 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:22.530 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:22.530 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:22.530 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:22.531 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:22.531 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:22.531 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:22.531 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:22.531 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:22.532 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.532 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:22.532 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:22.533 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:22.533 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:22.533 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:22.533 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:22.534 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:22.534 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:22.534 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:22.534 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:22.534 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:22.535 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.535 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:22.535 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:22.535 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:22.536 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:22.536 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:22.536 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:22.536 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:22.537 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:22.537 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:22.537 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:22.537 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:22.538 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.538 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:22.538 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:22.538 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:22.539 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:22.539 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:22.539 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:22.539 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:22.539 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:22.540 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:22.540 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:22.540 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:22.540 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.540 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.541 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:22.541 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:22.541 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:22.541 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:22.542 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:22.542 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:22.542 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:22.543 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:27.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:27.546 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:27.546 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:27.646 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:27.646 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:28.108 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:28.109 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:28.109 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:28.109 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:28.109 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:28.110 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:28.110 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:28.110 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:28.110 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:28.111 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:28.111 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:28.111 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:28.111 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:28.112 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:28.112 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:28.112 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:28.113 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:28.113 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:28.113 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:28.113 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.113 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:28.114 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:28.114 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:28.114 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:28.115 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:28.115 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:28.115 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:28.116 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:28.116 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:28.116 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:28.116 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:28.117 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.117 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:28.118 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:28.118 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:28.118 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:28.119 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:28.119 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:28.120 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:28.120 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:28.120 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:28.121 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:28.121 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:28.121 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.121 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:28.122 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:28.122 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:28.123 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:28.123 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:28.123 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:28.123 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:28.124 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:28.124 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:28.125 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:28.125 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:28.125 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.126 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.126 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:28.126 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:28.126 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:28.127 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:28.127 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:28.127 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:28.127 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:28.128 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:33.130 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:33.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:33.132 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:33.234 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:33.234 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:33.764 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:33.765 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:33.765 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:33.765 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:33.765 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:33.766 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:33.766 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:33.766 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:33.766 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:33.766 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:33.767 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:33.767 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:33.767 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:33.767 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:33.768 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:33.768 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:33.768 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:33.768 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:33.768 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:33.769 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.769 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:33.769 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:33.769 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:33.770 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:33.770 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:33.770 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:33.770 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:33.770 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:33.771 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:33.771 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:33.771 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:33.771 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.772 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:33.772 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:33.772 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:33.772 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:33.772 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:33.773 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:33.773 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:33.773 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:33.773 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:33.774 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:33.774 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:33.774 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.774 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:33.775 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:33.775 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:33.775 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:33.775 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:33.775 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:58:33.776 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:33.776 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:33.776 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:33.776 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:58:33.776 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:33.777 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.777 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.777 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:33.777 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:33.778 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:33.778 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:33.778 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:33.778 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:33.778 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:33.780 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T12:58:38.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:38.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:38.783 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:38.902 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:38.902 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:39.361 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:39.362 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:39.362 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:39.362 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:39.363 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:39.363 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:39.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:39.363 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:39.363 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:39.364 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:39.364 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:39.364 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:39.364 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:39.364 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:39.365 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:39.365 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:39.365 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:39.365 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:39.366 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:39.366 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.366 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:39.366 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:39.366 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:39.367 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:39.367 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:39.367 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:39.367 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:39.368 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:39.368 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:39.368 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:39.368 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:39.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.369 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:39.369 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:39.369 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:39.369 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:39.370 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:39.370 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:39.370 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:39.370 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:39.370 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:39.371 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:39.371 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:39.371 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.371 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:39.372 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:39.372 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:39.372 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:39.373 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:39.373 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T12:58:39.373 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:39.373 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:39.373 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:39.374 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T12:58:39.374 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:39.374 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.374 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.374 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:39.375 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:39.375 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:39.375 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:39.375 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:39.376 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:39.376 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:39.377 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.650194_182.1 2021-09-06T12:58:44.378 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:44.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:44.379 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:44.469 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:44.469 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:44.862 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:44.863 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:44.863 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:44.863 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:44.863 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:44.864 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:44.864 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:44.864 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:44.864 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:44.864 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:44.865 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:44.865 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:44.865 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:44.865 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:44.866 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:44.866 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:44.866 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:44.866 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:44.867 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:44.867 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.867 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:44.867 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:44.867 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:44.868 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:44.868 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:44.868 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:44.868 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:44.869 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:44.869 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:44.869 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:44.869 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:44.870 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.870 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:44.870 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:44.870 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:44.871 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:44.871 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:44.871 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:44.871 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:44.871 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:44.872 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:44.872 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:44.872 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:44.872 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.873 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:44.873 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:44.873 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:44.873 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:44.873 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:44.874 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T12:58:44.874 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:44.874 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:44.874 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:44.875 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T12:58:44.875 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:44.875 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.875 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.876 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:44.876 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:44.876 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:44.876 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:44.876 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:44.877 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:44.877 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:44.878 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T12:58:49.879 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:49.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:49.881 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:49.979 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:49.979 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:50.321 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:50.321 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:50.322 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:50.322 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:50.322 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:50.322 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:50.322 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:50.323 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:50.324 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:50.325 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:50.325 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:50.325 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:50.325 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.325 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:50.326 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:50.326 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:50.326 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:50.326 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:50.327 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:50.327 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:50.327 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:50.327 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:50.327 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:50.328 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:50.328 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.328 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:50.328 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:50.328 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:50.329 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:50.329 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:50.329 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:50.329 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:50.329 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:50.330 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:50.330 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:50.330 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:50.330 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.331 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:50.331 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:50.331 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:50.331 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:50.331 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:50.332 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:58:50.332 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:50.332 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:50.332 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:50.332 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:58:50.333 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:50.333 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.333 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.333 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:50.333 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:50.334 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:50.334 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:50.334 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:50.334 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:50.335 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:50.336 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:58:55.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:55.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:58:55.339 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:58:55.516 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:58:55.516 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:58:56.067 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:58:56.067 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:58:56.067 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:58:56.068 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:58:56.068 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:58:56.068 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:58:56.068 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:56.068 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:56.069 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:58:56.069 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:56.069 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:58:56.069 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:56.069 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:56.070 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:58:56.070 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:56.070 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:58:56.070 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:56.071 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:58:56.071 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:56.071 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.071 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:56.071 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:56.072 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:58:56.072 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:56.072 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:56.072 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:58:56.072 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:56.073 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:56.073 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:56.073 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:58:56.073 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:56.074 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.074 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:56.074 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:56.074 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:58:56.074 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:56.075 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:56.075 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:58:56.075 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:56.075 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:56.075 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:56.076 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:58:56.076 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:56.076 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.076 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:56.077 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:58:56.077 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:58:56.077 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:58:56.077 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:58:56.077 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:58:56.078 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:58:56.078 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:58:56.078 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:58:56.078 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:58:56.079 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:58:56.079 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.079 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.079 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:58:56.079 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:58:56.080 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:58:56.080 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:58:56.080 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:58:56.080 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:58:56.080 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:58:56.082 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T12:59:01.082 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:01.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:01.084 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:01.341 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:01.341 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:01.796 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:01.796 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:01.797 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:01.797 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:01.797 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:01.797 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:01.798 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:01.798 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:01.798 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:01.798 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:01.799 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:01.799 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:01.799 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:01.799 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:01.800 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:01.800 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:01.800 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:01.800 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:01.800 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:01.801 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.801 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:01.801 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:01.801 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:01.801 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:01.802 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:01.802 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:01.802 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:01.802 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:01.803 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:01.803 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:01.803 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:01.803 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.803 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:01.804 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:01.804 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:01.804 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:01.804 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:01.805 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:01.805 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:01.805 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:01.805 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:01.806 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:01.806 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:01.806 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.806 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:01.806 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:01.807 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:01.807 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:01.807 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:01.807 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T12:59:01.808 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:01.808 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:01.808 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:01.808 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T12:59:01.808 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:01.809 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.809 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.809 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:01.809 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:01.809 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:01.810 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:01.810 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:01.810 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:01.810 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:01.812 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T12:59:06.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:06.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:06.814 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:06.908 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:06.909 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:07.350 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:07.350 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:07.351 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:07.351 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:07.351 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:07.351 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:07.352 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:07.352 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:07.352 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:07.352 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:07.352 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:07.353 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:07.353 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:07.353 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:07.353 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:07.354 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:07.354 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:07.354 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:07.354 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:07.355 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.355 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:07.355 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:07.355 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:07.356 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:07.356 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:07.356 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:07.356 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:07.357 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:07.357 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:07.357 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:07.357 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:07.358 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.358 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:07.358 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:07.358 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:07.358 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:07.359 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:07.359 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:07.359 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:07.359 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:07.360 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:07.360 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:07.360 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:07.361 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.361 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:07.361 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:07.361 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:07.362 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:07.362 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:07.362 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:59:07.362 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:07.363 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:07.363 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:07.363 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:59:07.363 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:07.364 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.364 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.364 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:07.364 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:07.364 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:07.365 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:07.365 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:07.365 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:07.365 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:07.367 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:59:12.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:12.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:12.370 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:12.446 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:12.446 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:13.032 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:13.032 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:13.033 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:13.033 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:13.033 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:13.033 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:13.034 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:13.034 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:13.034 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:13.035 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:13.035 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:13.035 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:13.035 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:13.035 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:13.036 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:13.036 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:13.036 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:13.036 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:13.037 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.037 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:13.037 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:13.037 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:13.038 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:13.038 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:13.038 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:13.038 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:13.038 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:13.039 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:13.039 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:13.039 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:13.039 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.040 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:13.040 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:13.040 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:13.040 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:13.040 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:13.041 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:13.041 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:13.041 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:13.041 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:13.042 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:13.042 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:13.042 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.042 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:13.043 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:13.043 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:13.043 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:13.043 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:13.044 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T12:59:13.044 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:13.044 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:13.044 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:13.044 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T12:59:13.045 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:13.045 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.045 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.045 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:13.046 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:13.046 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:13.046 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:13.046 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:13.046 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:13.047 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:13.048 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T12:59:18.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:18.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:18.051 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:18.153 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:18.154 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:18.632 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:18.632 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:18.632 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:18.633 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:18.633 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:18.633 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:18.633 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:18.634 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:18.634 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:18.634 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:18.634 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:18.635 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:18.635 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:18.635 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:18.635 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:18.635 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:18.636 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:18.636 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:18.636 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:18.636 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.636 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:18.637 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:18.637 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:18.637 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:18.637 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:18.637 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:18.638 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:18.638 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:18.638 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:18.638 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:18.639 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:18.639 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.639 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:18.639 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:18.639 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:18.640 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:18.640 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:18.640 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:18.640 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:18.641 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:18.641 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:18.641 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:18.641 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:18.641 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.642 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:18.642 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:18.642 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:18.642 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:18.642 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:18.643 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T12:59:18.643 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:18.643 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:18.643 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:18.644 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T12:59:18.644 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:18.644 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.644 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.644 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:18.645 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:18.645 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:18.645 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:18.645 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:18.645 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:18.646 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:18.647 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T12:59:23.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:23.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:23.650 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:23.742 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:23.742 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:24.207 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:24.208 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:24.208 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:24.208 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:24.209 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:24.209 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:24.209 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:24.209 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:24.209 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:24.210 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:24.210 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:24.210 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:24.210 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:24.211 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:24.211 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:24.211 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:24.212 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:24.212 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:24.212 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:24.213 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.213 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:24.213 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:24.213 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:24.214 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:24.214 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:24.214 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:24.214 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:24.215 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:24.215 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:24.215 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:24.215 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:24.215 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.216 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:24.216 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:24.216 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:24.216 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:24.217 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:24.217 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:24.217 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:24.217 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:24.217 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:24.218 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:24.218 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:24.218 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.219 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:24.219 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:24.219 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:24.219 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:24.220 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:24.220 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T12:59:24.220 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:24.220 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:24.220 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:24.221 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T12:59:24.221 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:24.221 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.222 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.222 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:24.222 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:24.222 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:24.223 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:24.223 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:24.223 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:24.223 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:24.225 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T12:59:28.371 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T12:59:28.370+0000 7f04fd82c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T12:59:29.226 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:29.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:29.229 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:29.466 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:29.467 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:29.906 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:29.907 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:29.907 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:29.907 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:29.907 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:29.908 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:29.908 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:29.908 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:29.908 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:29.909 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:29.909 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:29.909 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:29.909 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:29.910 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:29.910 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:29.910 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:29.910 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:29.911 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:29.911 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:29.911 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.911 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:29.912 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:29.912 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:29.912 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:29.912 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:29.912 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:29.913 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:29.913 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:29.913 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:29.913 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:29.914 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:29.914 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.914 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:29.914 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:29.915 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:29.915 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:29.915 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:29.915 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:29.916 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:29.916 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:29.917 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:29.917 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:29.917 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:29.917 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.918 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:29.918 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:29.918 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:29.918 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:29.919 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:29.919 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:59:29.919 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:29.919 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:29.919 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:29.920 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:59:29.920 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:29.920 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.920 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.921 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:29.921 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:29.921 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:29.921 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:29.921 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:29.922 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:29.922 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:29.923 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T12:59:34.924 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:34.925 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:34.926 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:35.033 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:35.033 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:35.565 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:35.566 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:35.566 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:35.566 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:35.566 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:35.567 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:35.567 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:35.567 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:35.567 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:35.568 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:35.568 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:35.568 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:35.568 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:35.568 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:35.569 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:35.569 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:35.569 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:35.569 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:35.569 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:35.570 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.570 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:35.570 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:35.570 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:35.570 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:35.571 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:35.571 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:35.571 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:35.571 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:35.572 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:35.572 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:35.572 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:35.572 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.572 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:35.573 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:35.573 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:35.573 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:35.573 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:35.573 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:35.574 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:35.574 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:35.574 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:35.574 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:35.575 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:35.575 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.575 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:35.575 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:35.575 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:35.576 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:35.576 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:35.576 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:59:35.576 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:35.576 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:35.577 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:35.577 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:59:35.577 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:35.577 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.578 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.578 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:35.578 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:35.578 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:35.579 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:35.579 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:35.579 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:35.579 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:35.580 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T12:59:40.581 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:40.582 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:40.583 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:40.695 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:40.696 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:41.081 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:41.081 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:41.081 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:41.082 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:41.082 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:41.082 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:41.082 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:41.083 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:41.083 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:41.083 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:41.083 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:41.084 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:41.084 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:41.084 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:41.084 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:41.085 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:41.085 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:41.085 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:41.085 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:41.085 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.086 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:41.086 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:41.086 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:41.086 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:41.087 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:41.087 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:41.087 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:41.087 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:41.087 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:41.088 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:41.088 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:41.088 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.088 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:41.089 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:41.089 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:41.089 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:41.089 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:41.090 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:41.091 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:41.092 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:59:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:41.093 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:59:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.094 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:41.095 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:41.096 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:41.096 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:41.097 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T12:59:46.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:46.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:46.100 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:46.179 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:46.180 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:46.553 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:46.554 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:46.554 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:46.554 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:46.555 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:46.555 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:46.555 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:46.555 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:46.555 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:46.556 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:46.556 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:46.556 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:46.556 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:46.557 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:46.557 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:46.557 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:46.557 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:46.558 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:46.558 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:46.558 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.558 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:46.558 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:46.559 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:46.559 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:46.559 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:46.559 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:46.560 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:46.560 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:46.560 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:46.560 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:46.561 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:46.561 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.561 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:46.561 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:46.561 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:46.562 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:46.562 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:46.562 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:46.562 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:46.563 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:46.563 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:46.563 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:46.563 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:46.564 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.564 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:46.564 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:46.564 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:46.565 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:46.565 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:46.565 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T12:59:46.565 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:46.565 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:46.566 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:46.566 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T12:59:46.566 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:46.566 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.567 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.567 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:46.567 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:46.567 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:46.568 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:46.568 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:46.568 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:46.568 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:46.569 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T12:59:51.570 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:51.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:51.572 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:51.671 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:51.672 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:52.317 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:52.317 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:52.317 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:52.318 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:52.318 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:52.318 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:52.319 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:52.319 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:52.320 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:52.320 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:52.321 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:52.321 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:52.321 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:52.322 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:52.322 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:52.322 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:52.323 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:52.323 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:52.324 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:52.324 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.325 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:52.325 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:52.326 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:52.326 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:52.327 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:52.327 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:52.328 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:52.328 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:52.329 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:52.329 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:52.330 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:52.330 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.330 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:52.331 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:52.331 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:52.332 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:52.332 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:52.333 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:52.333 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:52.333 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:52.334 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:52.334 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:52.335 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:52.335 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.335 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:52.336 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:52.336 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:52.337 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:52.337 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:52.338 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T12:59:52.338 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:52.339 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:52.339 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:52.339 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T12:59:52.340 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:52.340 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.340 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.341 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:52.341 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:52.341 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:52.341 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:52.342 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:52.342 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:52.342 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:52.344 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T12:59:57.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:57.346 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T12:59:57.348 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T12:59:57.444 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T12:59:57.445 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T12:59:57.863 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T12:59:57.863 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T12:59:57.863 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T12:59:57.864 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T12:59:57.864 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T12:59:57.864 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T12:59:57.864 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:57.864 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:57.865 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T12:59:57.865 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:57.865 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T12:59:57.865 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:57.866 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:57.866 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T12:59:57.866 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:57.866 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T12:59:57.866 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:57.867 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T12:59:57.867 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:57.867 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.867 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:57.868 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:57.868 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T12:59:57.868 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:57.868 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:57.868 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T12:59:57.869 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:57.869 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:57.869 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:57.869 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T12:59:57.870 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:57.870 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.870 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:57.870 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:57.871 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T12:59:57.871 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:57.871 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:57.871 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T12:59:57.871 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:57.872 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:57.872 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:57.872 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T12:59:57.872 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:57.873 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.873 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:57.873 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T12:59:57.873 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T12:59:57.873 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T12:59:57.874 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T12:59:57.874 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T12:59:57.874 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T12:59:57.874 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T12:59:57.875 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T12:59:57.875 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T12:59:57.875 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T12:59:57.875 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.875 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.876 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T12:59:57.876 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T12:59:57.876 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T12:59:57.876 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T12:59:57.876 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T12:59:57.877 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T12:59:57.877 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T12:59:57.878 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:00:02.880 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:02.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:02.881 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:02.968 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:02.968 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:03.360 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:03.360 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:03.360 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:03.361 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:03.361 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:03.361 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:03.362 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:03.362 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:03.363 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:03.363 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:03.363 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:03.363 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:03.364 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:03.364 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:03.364 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:03.364 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:03.365 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:03.365 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:03.365 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:03.365 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.366 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:03.366 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:03.366 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:03.366 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:03.366 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:03.367 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:03.367 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:03.367 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:03.368 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:03.368 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:03.368 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:03.368 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.368 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:03.369 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:03.369 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:03.369 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:03.369 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:03.370 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:03.370 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:03.370 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:03.370 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:03.371 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:03.371 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:03.371 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.371 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:03.372 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:03.372 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:03.372 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:03.373 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:03.373 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:00:03.373 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:03.374 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:03.374 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:03.374 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:00:03.374 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:03.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.375 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.375 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:03.375 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:03.376 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:03.376 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:03.376 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:03.376 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:03.376 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:03.378 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:00:08.379 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:08.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:08.381 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:08.474 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:08.475 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:08.944 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:08.944 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:08.945 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:08.945 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:08.945 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:08.945 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:08.946 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:08.946 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:08.946 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:08.947 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:08.947 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:08.947 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:08.947 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:08.948 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:08.948 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:08.948 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:08.948 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:08.948 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:08.949 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:08.949 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.949 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:08.949 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:08.950 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:08.950 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:08.950 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:08.950 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:08.950 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:08.951 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:08.951 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:08.951 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:08.951 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:08.952 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.952 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:08.952 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:08.952 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:08.952 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:08.953 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:08.953 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:08.953 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:08.953 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:08.954 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:08.954 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:08.955 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:08.955 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.955 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:08.955 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:08.956 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:08.956 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:08.956 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:08.956 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:00:08.957 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:08.957 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:08.957 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:08.957 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:00:08.957 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:08.958 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.958 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.958 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:08.958 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:08.959 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:08.959 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:08.959 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:08.959 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:08.960 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:08.960 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:00:13.961 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:13.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:13.963 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:14.057 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:14.058 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:14.604 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:14.604 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:14.604 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:14.605 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:14.605 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:14.605 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:14.605 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:14.606 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:14.606 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:14.606 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:14.606 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:14.606 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:14.607 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:14.607 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:14.607 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:14.607 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:14.608 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:14.608 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:14.608 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:14.608 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.609 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:14.609 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:14.609 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:14.609 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:14.609 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:14.610 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:14.610 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:14.610 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:14.610 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:14.611 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:14.611 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:14.611 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.611 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:14.612 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:14.612 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:14.612 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:14.612 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:14.613 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:14.613 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:14.613 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:14.614 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:14.614 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:14.614 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:14.615 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.615 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:14.615 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:14.616 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:14.616 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:14.616 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:14.616 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:00:14.617 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:14.617 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:14.617 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:14.617 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:00:14.617 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:14.618 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.618 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.618 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:14.618 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:14.619 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:14.619 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:14.619 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:14.619 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:14.620 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:14.621 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T13:00:19.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:19.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:19.624 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:19.763 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:19.764 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:20.156 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:20.156 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:20.156 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:20.157 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:20.157 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:20.157 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:20.157 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:20.157 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:20.158 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:20.158 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:20.158 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:20.158 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:20.159 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:20.159 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:20.159 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:20.159 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:20.159 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:20.160 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:20.160 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:20.160 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.160 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:20.160 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:20.161 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:20.161 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:20.161 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:20.161 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:20.162 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:20.162 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:20.162 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:20.162 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:20.162 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:20.163 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.163 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:20.164 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:20.164 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:20.164 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:20.164 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:20.164 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:20.165 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:20.165 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:20.165 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:20.165 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:20.166 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:20.166 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.166 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:20.166 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:20.166 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:20.167 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:20.167 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:20.167 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:00:20.167 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:20.167 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:20.168 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:20.168 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:00:20.168 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:20.168 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.169 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.169 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:20.169 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:20.169 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:20.169 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:20.170 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:20.170 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:20.170 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:20.171 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:00:25.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:25.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:25.174 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:25.258 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:25.259 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:25.828 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:25.828 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:25.829 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:25.829 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:25.829 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:25.829 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:25.830 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:25.830 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:25.830 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:25.830 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:25.831 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:25.831 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:25.831 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:25.831 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:25.832 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:25.832 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:25.832 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:25.832 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:25.832 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:25.833 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.833 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:25.833 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:25.833 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:25.834 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:25.834 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:25.834 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:25.834 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:25.835 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:25.835 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:25.835 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:25.835 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:25.835 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.836 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:25.836 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:25.836 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:25.836 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:25.837 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:25.837 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:25.837 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:25.837 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:25.837 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:25.838 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:25.838 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:25.838 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.838 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:25.839 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:25.839 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:25.839 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:25.839 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:25.840 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:00:25.840 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:25.840 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:25.840 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:25.840 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:00:25.841 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:25.841 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.841 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.841 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:25.842 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:25.842 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:25.842 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:25.842 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:25.842 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:25.843 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:25.844 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:00:30.845 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:30.846 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:30.847 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:30.949 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:30.950 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:31.432 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:31.433 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:31.433 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:31.433 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:31.433 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:31.434 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:31.434 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:31.434 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:31.434 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:31.434 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:31.435 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:31.435 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:31.435 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:31.435 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:31.436 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:31.436 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:31.436 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:31.436 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:31.441 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:31.441 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.441 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:31.441 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:31.442 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:31.442 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:31.442 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:31.442 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:31.443 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:31.443 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:31.443 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:31.443 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:31.443 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:31.444 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.444 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:31.444 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:31.444 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:31.445 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:31.445 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:31.445 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:31.445 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:31.446 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:31.446 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:31.446 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:31.446 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:31.447 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.447 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:31.447 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:31.447 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:31.447 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:31.448 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:31.448 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:00:31.448 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:31.448 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:31.449 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:31.449 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:00:31.449 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:31.449 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.450 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.450 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:31.450 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:31.450 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:31.450 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:31.451 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:31.451 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:31.451 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:31.452 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T13:00:36.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:36.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:36.455 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:36.578 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:36.579 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:36.952 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:36.952 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:36.953 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:36.953 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:36.953 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:36.953 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:36.954 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:36.954 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:36.954 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:36.954 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:36.955 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:36.955 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:36.955 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:36.955 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:36.955 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:36.956 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:36.956 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:36.956 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:36.956 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:36.957 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.957 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:36.957 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:36.957 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:36.958 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:36.958 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:36.958 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:36.958 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:36.959 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:36.959 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:36.959 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:36.959 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:36.960 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.960 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:36.960 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:36.960 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:36.961 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:36.961 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:36.961 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:36.961 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:36.961 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:36.962 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:36.962 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:36.962 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:36.963 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.963 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:36.963 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:36.963 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:36.964 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:36.964 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:36.964 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:00:36.964 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:36.965 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:36.965 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:36.965 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:00:36.965 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:36.965 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.966 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.966 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:36.966 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:36.966 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:36.966 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:36.967 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:36.967 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:36.967 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:36.969 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T13:00:41.971 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:41.972 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:41.972 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:42.055 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:42.055 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:42.672 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:42.672 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:42.673 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:42.673 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:42.673 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:42.673 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:42.674 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:42.674 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:42.674 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:42.674 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:42.674 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:42.675 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:42.675 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:42.675 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:42.675 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:42.676 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:42.676 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:42.676 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:42.676 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:42.677 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.677 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:42.677 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:42.677 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:42.677 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:42.678 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:42.678 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:42.678 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:42.678 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:42.679 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:42.679 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:42.679 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:42.679 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.679 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:42.680 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:42.680 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:42.680 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:42.680 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:42.681 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:42.681 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:42.681 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:42.681 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:42.681 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:42.682 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:42.682 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.682 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:42.682 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:42.683 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:42.683 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:42.683 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:42.683 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:00:42.683 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:42.684 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:42.684 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:42.684 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:00:42.684 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:42.684 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.685 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.685 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:42.685 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:42.685 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:42.685 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:42.686 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:42.686 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:42.686 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:42.687 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:00:47.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:47.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:47.690 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:47.768 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:47.769 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:48.435 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:48.435 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:48.436 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:48.436 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:48.436 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:48.437 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:48.437 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:48.437 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:48.437 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:48.437 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:48.438 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:48.438 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:48.438 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:48.438 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:48.439 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:48.439 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:48.439 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:48.439 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:48.439 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:48.440 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.440 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:48.440 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:48.440 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:48.441 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:48.441 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:48.441 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:48.441 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:48.441 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:48.442 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:48.442 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:48.442 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:48.442 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.443 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:48.443 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:48.443 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:48.443 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:48.444 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:48.444 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:48.444 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:48.444 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:48.444 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:48.445 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:48.445 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:48.445 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.445 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:48.446 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:48.446 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:48.446 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:48.446 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:48.446 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:00:48.447 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:48.447 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:48.447 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:48.447 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:00:48.448 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:48.448 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.448 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.448 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:48.448 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:48.449 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:48.449 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:48.449 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:48.449 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:48.450 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:48.451 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:00:53.453 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:53.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:53.455 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:53.880 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:53.881 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:54.063 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:54.064 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:54.064 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:54.064 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:54.064 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:54.065 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:54.065 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:54.065 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:54.065 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:54.065 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:54.066 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:54.066 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:54.066 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:54.066 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:54.067 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:54.067 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:54.067 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:54.067 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:54.068 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:54.068 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.068 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:54.068 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:54.068 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:54.069 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:54.069 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:54.069 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:54.070 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:54.070 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:54.070 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:54.070 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:54.071 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:54.071 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.071 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:54.071 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:54.071 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:54.072 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:54.072 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:54.072 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:54.072 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:54.073 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:54.073 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:54.073 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:54.073 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:54.074 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.074 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:54.074 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:54.074 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:54.074 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:54.075 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:54.075 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:00:54.075 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:54.075 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:54.076 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:54.076 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:00:54.076 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:54.076 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.077 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.077 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:54.077 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:54.078 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:54.078 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:54.078 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:54.078 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:54.079 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:54.080 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:00:59.081 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:59.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:00:59.082 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:00:59.159 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:00:59.159 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:00:59.555 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:00:59.555 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:00:59.555 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:00:59.556 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:00:59.556 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:00:59.556 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:00:59.556 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:59.556 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:59.557 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:00:59.557 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:59.557 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:00:59.557 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:59.558 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:59.558 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:00:59.558 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:59.558 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:00:59.558 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:59.559 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:00:59.559 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:59.559 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.559 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:59.560 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:59.560 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:00:59.560 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:59.560 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:59.560 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:00:59.561 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:59.561 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:59.561 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:59.561 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:00:59.562 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:59.562 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.562 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:59.562 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:59.562 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:00:59.563 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:59.563 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:59.563 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:00:59.563 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:59.564 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:59.564 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:59.564 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:00:59.564 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:59.564 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.565 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:59.565 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:00:59.565 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:00:59.565 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:00:59.565 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:00:59.566 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:00:59.566 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:00:59.566 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:00:59.566 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:00:59.567 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:00:59.567 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:00:59.567 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.567 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.567 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:00:59.568 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:00:59.568 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:00:59.568 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:00:59.568 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:00:59.569 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:00:59.569 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:00:59.570 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:01:04.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:04.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:04.573 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:04.659 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:04.660 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:05.203 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:05.204 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:05.204 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:05.204 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:05.204 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:05.204 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:05.205 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:05.205 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:05.205 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:05.205 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:05.206 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:05.206 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:05.206 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:05.206 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:05.206 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:05.207 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:05.207 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:05.207 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:05.207 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:05.208 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.208 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:05.208 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:05.208 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:05.208 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:05.209 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:05.209 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:05.209 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:05.209 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:05.209 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:05.210 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:05.210 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:05.210 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.210 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:05.210 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:05.211 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:05.211 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:05.211 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:05.211 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:05.212 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:05.212 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:05.212 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:05.213 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:05.213 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:05.213 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.213 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:05.213 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:05.214 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:05.214 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:05.214 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:05.214 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:01:05.215 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:05.215 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:05.215 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:05.215 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:01:05.216 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:05.216 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.216 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.216 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:05.216 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:05.217 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:05.217 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:05.217 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:05.217 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:05.217 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:05.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.595266_180.1 2021-09-06T13:01:10.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:10.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:10.221 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:10.320 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:10.321 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:10.775 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:10.775 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:10.775 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:10.776 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:10.776 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:10.776 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:10.776 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:10.777 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:10.777 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:10.777 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:10.777 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:10.777 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:10.778 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:10.778 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:10.778 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:10.778 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:10.779 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:10.779 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:10.779 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:10.779 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.780 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:10.780 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:10.780 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:10.780 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:10.780 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:10.781 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:10.781 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:10.781 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:10.781 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:10.782 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:10.782 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:10.782 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.782 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:10.783 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:10.783 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:10.783 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:10.783 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:10.783 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:10.784 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:10.784 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:10.784 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:10.784 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:10.785 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:10.785 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.785 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:10.785 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:10.785 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:10.786 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:10.786 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:10.786 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:01:10.786 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:10.787 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:10.787 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:10.787 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:01:10.787 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:10.788 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.788 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.788 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:10.788 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:10.789 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:10.789 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:10.789 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:10.789 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:10.789 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:10.790 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:01:15.791 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:15.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:15.792 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:15.864 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:15.865 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:16.229 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:16.229 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:16.229 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:16.230 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:16.230 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:16.230 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:16.230 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:16.231 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:16.231 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:16.231 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:16.231 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:16.232 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:16.232 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:16.232 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:16.232 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:16.232 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:16.233 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:16.233 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:16.233 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:16.233 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.234 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:16.234 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:16.234 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:16.234 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:16.234 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:16.235 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:16.235 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:16.235 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:16.236 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:16.236 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:16.236 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:16.236 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.236 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:16.237 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:16.237 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:16.237 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:16.237 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:16.238 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:16.238 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:16.238 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:16.238 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:16.239 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:16.239 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:16.239 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.239 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:16.239 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:16.240 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:16.240 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:16.240 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:16.240 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:01:16.241 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:16.241 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:16.241 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:16.241 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:01:16.241 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:16.242 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.242 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.242 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:16.242 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:16.243 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:16.243 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:16.243 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:16.243 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:16.243 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:16.245 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:01:21.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:21.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:21.248 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:21.344 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:21.345 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:21.893 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:21.893 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:21.894 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:21.894 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:21.894 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:21.894 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:21.895 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:21.895 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:21.895 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:21.895 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:21.896 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:21.896 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:21.896 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:21.896 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:21.897 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:21.897 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:21.897 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:21.897 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:21.897 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:21.898 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.898 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:21.898 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:21.898 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:21.899 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:21.899 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:21.899 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:21.899 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:21.900 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:21.900 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:21.900 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:21.900 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:21.901 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.901 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:21.901 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:21.901 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:21.901 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:21.902 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:21.902 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:21.902 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:21.902 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:21.903 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:21.903 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:21.903 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:21.903 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.903 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:21.904 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:21.904 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:21.904 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:21.904 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:21.905 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:01:21.905 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:21.905 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:21.905 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:21.905 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:01:21.906 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:21.906 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.906 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.906 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:21.907 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:21.907 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:21.907 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:21.907 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:21.907 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:21.908 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:21.908 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:01:26.909 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:26.911 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:26.912 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:27.013 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:27.014 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:27.322 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:27.323 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:27.323 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:27.323 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:27.324 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:27.324 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:27.324 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:27.324 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:27.325 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:27.325 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:27.325 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:27.325 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:27.325 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:27.326 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:27.326 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:27.326 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:27.326 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:27.326 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:27.327 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:27.327 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.327 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:27.327 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:27.328 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:27.328 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:27.328 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:27.328 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:27.328 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:27.329 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:27.329 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:27.329 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:27.329 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:27.330 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.330 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:27.330 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:27.330 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:27.331 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:27.331 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:27.331 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:27.331 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:27.332 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:27.332 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:27.332 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:27.332 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:27.332 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.333 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:27.333 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:27.333 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:27.333 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:27.334 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:27.334 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:01:27.334 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:27.334 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:27.334 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:27.335 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:01:27.335 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:27.335 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.335 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.335 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:27.336 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:27.336 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:27.336 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:27.336 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:27.337 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:27.337 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:27.338 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:01:32.339 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:32.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:32.341 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:32.435 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:32.436 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:32.879 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:32.880 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:32.880 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:32.880 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:32.880 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:32.881 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:32.881 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:32.881 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:32.882 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:32.882 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:32.883 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:32.883 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:32.884 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:32.884 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:32.884 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:32.885 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:32.885 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:32.885 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:32.885 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:32.886 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.886 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:32.886 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:32.886 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:32.887 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:32.887 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:32.887 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:32.887 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:32.888 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:32.888 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:32.888 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:32.888 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:32.888 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.889 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:32.889 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:32.889 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:32.889 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:32.890 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:32.890 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:32.890 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:32.891 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:32.891 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:32.891 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:32.891 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:32.891 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.892 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:32.892 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:32.892 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:32.893 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:32.893 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:32.893 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:01:32.893 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:32.893 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:32.894 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:32.894 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:01:32.894 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:32.894 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.895 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.895 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:32.896 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:32.896 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:32.896 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:32.897 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:32.897 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:32.898 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:32.900 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:01:37.901 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:37.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:37.903 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:37.992 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:37.992 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:38.497 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:38.497 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:38.497 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:38.498 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:38.498 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:38.498 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:38.498 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:38.499 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:38.499 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:38.499 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:38.499 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:38.500 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:38.500 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:38.500 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:38.500 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:38.500 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:38.501 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:38.501 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:38.501 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:38.501 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.502 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:38.502 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:38.502 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:38.502 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:38.503 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.504 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:38.505 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:38.506 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.507 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:38.508 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:01:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:38.509 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:01:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:38.510 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:38.511 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:38.512 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:38.513 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:01:43.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:43.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:43.515 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:43.568 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:43.568 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:43.659 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:01:43.660 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:01:43.660 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:01:43.660 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:01:43.661 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:01:43.661 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:43.661 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:43.661 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:43.661 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:01:43.662 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:43.662 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:01:43.662 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:43.662 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:43.663 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:43.663 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:43.663 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:01:43.663 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:43.664 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:43.664 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:43.664 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.664 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:43.665 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:43.665 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:01:43.665 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:43.665 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:43.665 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:01:43.666 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:43.666 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:43.666 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:43.666 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:01:43.667 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:43.667 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.667 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:43.667 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:43.667 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:01:43.668 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:43.668 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:43.668 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:43.668 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:43.669 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:43.669 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:43.669 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:43.669 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:43.669 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.670 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:43.670 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:01:43.670 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:01:43.670 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:01:43.671 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:01:43.671 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630933021.264676_258.1", 2021-09-06T13:01:43.671 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:01:43.671 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:01:43.672 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:01:43.672 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:57:01.264676Z", 2021-09-06T13:01:43.672 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:01:43.672 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.672 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.673 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:01:43.673 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:01:43.673 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:01:43.673 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:01:43.674 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:01:43.674 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:01:43.674 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:01:43.675 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T13:01:43.676 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T13:01:43.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:43.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:43.677 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:44.277 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:44.277 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:44.691 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:01:44.692 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:01:44.692 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:01:44.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:01:44.692 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:01:44.693 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:44.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:44.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:44.693 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:01:44.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:44.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:01:44.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:44.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:44.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:44.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:44.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:01:44.695 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:44.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:44.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:44.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:44.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:44.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:01:44.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:44.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:44.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:01:44.697 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:44.698 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:44.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:44.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:01:44.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:44.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:44.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:44.699 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:01:44.699 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:44.700 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:44.700 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:44.701 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:44.701 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:44.701 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:44.701 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:44.702 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:44.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:44.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:44.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:01:44.703 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:44.703 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:44.703 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:01:44.703 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:44.704 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:44.704 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:44.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:01:44.704 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:44.704 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.705 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:01:44.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:44.705 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:01:44.706 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:01:44.706 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:01:44.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:44.706 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:01:44.708 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.516029_175.1 2021-09-06T13:01:49.236 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T13:01:49.237+0000 7f04fc029700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T13:01:49.244 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T13:01:49.244+0000 7f89dff66700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T13:01:49.709 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:49.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:49.711 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:49.807 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:49.807 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:50.057 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:01:50.058 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:01:50.058 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:01:50.058 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:01:50.058 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:01:50.059 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:50.059 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:50.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:50.059 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:01:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:01:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:01:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:01:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:01:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:01:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:01:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:01:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:01:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:01:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:01:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:01:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:01:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:01:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:50.071 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:01:50.073 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:01:55.074 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:55.075 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:01:55.076 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:01:55.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:01:55.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:01:55.314 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:01:55.314 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:01:55.315 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:01:55.315 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:01:55.315 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:01:55.315 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:01:55.316 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:55.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:55.316 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:01:55.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:55.316 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:01:55.317 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:55.317 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:55.317 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:01:55.317 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:55.317 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:01:55.318 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:55.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:01:55.318 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:55.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:55.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:55.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:01:55.319 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:55.319 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:55.320 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:01:55.320 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:55.320 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:55.320 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:55.320 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:01:55.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:55.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.321 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:55.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:55.322 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:01:55.322 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:55.322 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:55.322 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:01:55.322 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:55.323 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:55.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:55.323 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:01:55.323 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:55.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:55.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:01:55.324 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:01:55.324 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:01:55.325 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:01:55.325 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:01:55.325 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:01:55.325 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:01:55.325 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:01:55.326 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:01:55.326 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:01:55.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.327 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:01:55.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:01:55.327 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:01:55.327 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:01:55.327 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:01:55.328 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:01:55.328 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:01:55.329 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T13:02:00.330 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:00.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:00.332 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:00.438 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:00.439 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:00.806 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:00.806 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:00.806 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:00.807 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:00.807 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:00.807 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:00.807 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:00.808 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:00.808 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:00.808 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:00.808 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:00.809 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:00.809 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:00.809 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:00.809 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:00.809 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:00.810 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:00.810 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:00.810 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:00.810 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:00.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:00.811 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:00.811 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:00.812 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:00.812 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:00.812 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:00.812 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:00.812 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:00.813 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:00.813 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:00.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:00.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:00.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:00.814 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:00.814 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:00.814 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:00.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:00.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:00.815 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:00.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:00.815 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:00.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:00.817 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:02:00.818 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:00.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.818 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:00.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:00.820 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:00.821 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:02:05.822 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:05.822 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:05.823 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:05.917 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:05.918 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:06.362 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:06.362 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:06.362 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:06.363 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:06.363 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:06.363 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:06.363 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:06.363 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:06.364 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:06.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:06.364 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:06.364 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:06.365 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:06.365 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:06.365 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:06.365 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:06.365 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:06.366 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:06.366 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:06.366 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:06.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:06.367 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:06.367 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:06.367 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:06.367 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:06.368 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:06.368 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:06.368 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:06.368 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:06.369 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:06.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:06.369 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:06.369 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:06.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:06.370 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:06.370 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:06.370 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:06.371 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:06.371 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:06.371 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:06.371 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:06.371 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:06.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:06.372 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:06.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:06.373 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:06.373 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:02:06.373 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:06.373 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:06.373 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:06.374 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:02:06.374 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:06.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.374 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:06.375 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:06.375 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:06.375 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:06.375 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:06.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:06.376 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:06.376 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T13:02:11.377 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:11.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:11.378 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:11.482 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:11.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:11.756 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:11.756 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:11.757 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:11.758 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:11.759 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:11.761 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:11.762 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:11.763 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:11.764 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:11.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:11.766 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:02:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:02:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.768 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:11.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:11.770 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:11.770 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:02:16.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:16.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:16.773 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:16.874 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:16.875 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:17.420 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:17.420 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:17.421 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:17.422 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:17.423 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:17.424 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:17.425 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:17.426 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:17.427 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:17.428 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:17.429 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:17.430 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.431 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:17.432 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:02:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:02:17.433 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:17.434 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:17.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:17.435 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:17.436 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T13:02:22.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:22.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:22.439 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:22.529 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:22.864 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:22.865 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:22.865 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:22.866 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:22.867 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:22.868 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:22.869 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:22.870 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:22.871 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:22.872 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:22.873 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:22.874 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:22.875 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:22.876 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:22.877 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:02:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:22.878 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:02:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.879 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:22.880 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:22.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:22.881 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:22.882 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:02:27.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:27.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:27.885 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:27.966 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:27.966 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:28.303 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:28.304 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:28.304 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:28.304 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:28.305 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:28.305 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:28.305 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:28.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:28.306 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:28.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:28.306 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:28.306 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:28.307 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:28.307 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:28.307 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:28.307 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:28.308 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:28.308 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:28.308 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:28.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:28.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:28.309 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:28.309 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:28.309 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:28.310 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:28.310 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:28.310 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:28.310 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:28.311 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:28.311 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:28.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:28.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:28.312 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:28.312 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:28.312 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:28.312 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:28.313 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:28.313 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:28.313 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:28.314 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:28.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:28.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:28.315 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:28.315 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:28.315 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:28.316 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:28.316 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:02:28.316 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:28.316 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:28.317 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:28.317 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:02:28.317 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:28.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.318 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.318 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:28.318 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:28.318 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:28.319 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:28.319 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:28.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:28.319 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:28.321 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:02:33.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:33.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:33.323 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:33.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:33.424 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:33.639 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:33.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:33.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:33.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:33.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:33.644 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:33.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:33.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:33.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:33.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:33.645 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:33.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:33.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:33.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:33.646 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:33.646 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:33.646 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:33.646 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:33.647 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:33.647 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:33.647 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:33.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:33.647 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:33.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:33.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:33.648 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:33.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:33.649 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:33.649 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:02:33.649 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:33.649 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:33.650 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:33.650 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:02:33.650 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:33.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.651 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:33.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:33.651 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:33.651 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:33.651 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:33.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:33.652 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:33.653 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.650194_182.1 2021-09-06T13:02:38.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:38.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:38.656 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:38.755 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:38.756 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:39.000 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:39.001 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:39.001 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:39.001 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:39.002 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:39.002 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:39.002 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:39.002 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:39.003 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:39.003 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:39.003 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:39.003 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:39.004 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:39.004 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:39.004 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:39.004 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:39.005 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:39.005 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:39.005 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:39.005 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.005 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:39.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:39.006 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:39.006 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:39.006 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:39.007 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:39.008 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:39.009 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:39.010 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:39.011 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:02:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:39.012 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:02:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:39.013 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:39.014 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:39.015 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:39.015 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:39.016 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.542715_178.1 2021-09-06T13:02:44.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:44.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:44.020 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:44.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:44.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:44.500 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:44.500 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:44.501 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:44.502 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:44.503 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:44.504 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:44.506 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:44.507 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:44.508 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:44.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:44.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:44.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:44.512 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:44.513 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:44.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:44.515 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:02:44.516 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:44.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:02:44.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:44.520 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:44.521 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:44.522 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:44.522 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:44.524 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:02:49.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:49.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:49.527 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:49.630 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:49.631 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:50.059 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:50.060 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:50.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:50.062 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:50.063 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:50.064 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:50.065 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:50.066 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:50.067 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:50.068 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:50.069 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:02:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:02:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:50.073 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:50.074 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T13:02:55.075 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:55.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:02:55.076 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:02:55.122 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:02:55.122 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:02:55.679 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:02:55.680 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:02:55.680 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:02:55.680 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:02:55.681 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:02:55.681 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:02:55.681 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:55.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:55.682 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:02:55.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:55.682 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:02:55.682 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:55.683 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:55.683 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:02:55.683 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:55.683 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:02:55.683 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:55.684 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:02:55.684 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:55.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:55.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:55.685 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:02:55.685 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:55.685 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:55.685 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:02:55.686 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:55.686 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:55.686 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:55.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:02:55.687 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:55.687 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:55.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:55.687 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:02:55.688 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:55.688 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:55.688 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:02:55.688 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:55.689 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:55.689 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:55.689 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:02:55.689 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:55.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:55.690 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:02:55.690 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:02:55.690 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:02:55.691 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:02:55.691 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:02:55.691 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:02:55.691 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:02:55.692 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:02:55.692 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:02:55.692 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:02:55.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.693 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:02:55.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:02:55.693 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:02:55.693 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:02:55.694 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:02:55.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:02:55.694 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:02:55.695 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:03:00.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:00.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:00.698 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:00.816 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:00.817 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:01.133 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:01.133 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:01.134 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:01.135 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:01.136 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:01.137 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:01.138 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:01.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:01.140 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:01.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:01.142 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:03:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:01.143 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:03:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:01.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:01.146 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:01.147 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:03:06.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:06.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:06.150 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:06.312 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:06.313 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:06.612 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:06.612 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:06.613 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:06.614 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:06.615 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:06.616 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:06.617 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:06.618 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:06.620 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:06.621 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:06.622 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:03:06.623 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:03:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:06.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:06.625 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:06.626 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:06.626 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:06.626 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:06.627 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:03:11.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:11.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:11.630 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:12.496 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:12.497 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:12.740 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:12.741 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:12.741 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:12.741 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:12.741 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:12.741 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:12.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:12.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:12.742 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:12.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:12.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:12.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:12.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:12.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:12.743 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:12.744 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:12.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:12.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:12.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:12.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:12.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:12.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:12.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:12.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:12.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:12.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:12.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:12.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:12.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:12.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:12.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:12.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:12.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:12.749 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:12.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:12.749 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:12.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:12.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:12.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:12.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:12.750 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:12.751 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:03:12.751 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:12.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:12.751 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:12.751 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:03:12.752 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:12.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.752 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:12.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:12.754 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:12.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.523913_177.1 2021-09-06T13:03:17.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:17.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:17.758 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:17.817 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:17.817 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:18.274 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:18.274 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:18.275 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:18.275 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:18.275 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:18.275 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:18.276 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:18.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:18.276 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:18.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:18.277 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:18.277 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:18.277 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:18.277 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:18.277 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:18.278 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:18.278 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:18.278 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:18.278 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:18.278 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:18.279 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:18.279 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:18.279 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:18.280 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:18.280 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:18.280 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:18.280 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:18.281 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:18.281 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:18.281 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:18.281 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.281 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:18.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:18.282 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:18.282 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:18.282 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:18.283 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:18.283 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:18.283 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:18.283 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:18.283 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:18.284 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:18.284 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.284 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:18.284 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:18.285 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:18.285 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:18.285 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:18.285 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:03:18.285 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:18.286 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:18.286 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:18.286 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:03:18.286 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:18.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.287 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:18.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:18.287 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:18.288 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:18.288 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:18.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:18.288 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:18.289 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T13:03:23.290 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:23.291 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:23.292 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:23.364 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:23.365 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:23.636 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:23.645 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:23.645 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:23.645 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:23.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:23.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:23.646 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:23.646 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:23.647 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:23.647 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:23.647 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:03:23.647 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:23.648 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:23.648 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:23.648 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:03:23.648 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:23.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.649 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:23.649 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:23.649 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:23.649 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:23.650 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:23.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:23.650 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:23.651 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:03:28.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:28.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:28.655 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:28.734 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:28.735 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:28.964 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:28.965 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:28.965 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:28.965 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:28.965 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:28.966 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:28.966 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:28.966 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:28.966 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:28.967 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:28.967 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:28.967 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:28.967 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:28.968 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:28.969 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:28.970 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:28.971 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:28.972 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:28.973 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:28.974 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:28.975 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:03:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:03:28.976 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:28.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:28.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:28.979 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:28.980 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T13:03:33.981 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:33.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:33.983 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:34.063 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:34.064 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:34.259 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:34.261 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:34.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:34.264 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:34.265 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:34.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:34.267 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:34.268 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:34.268 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:34.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.268 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:34.268 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:34.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:03:34.270 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:03:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:34.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:34.272 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:34.273 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:34.273 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:34.274 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.517790_176.1 2021-09-06T13:03:39.276 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:39.276 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:39.277 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:39.329 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:39.330 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:39.686 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:39.687 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:39.687 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:39.687 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:39.688 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:39.688 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:39.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:39.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:39.689 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:39.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:39.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:39.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:39.690 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:39.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:39.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:39.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:39.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:39.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:39.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:39.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:39.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:39.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:39.692 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:39.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:39.693 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:39.693 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:39.693 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:39.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:39.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:39.694 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:39.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:39.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:39.695 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:39.695 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:39.695 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:39.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:39.696 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:39.696 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:39.696 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:39.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:39.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:39.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:39.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:39.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:39.698 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:39.698 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:39.698 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:03:39.698 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:39.698 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:39.699 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:39.699 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:03:39.699 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:39.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.700 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:39.700 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:39.700 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:39.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:39.701 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:39.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:39.701 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:39.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:03:44.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:44.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:44.705 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:44.748 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:44.749 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:44.873 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:44.873 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:44.873 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:44.874 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:44.874 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:44.874 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:44.874 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:44.875 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:44.875 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:44.875 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:44.875 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:44.876 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:44.876 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:44.876 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:44.876 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:44.877 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:44.877 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:44.877 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:44.877 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:44.877 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:44.878 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:44.878 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:44.878 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:44.879 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:44.879 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:44.879 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:44.879 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:44.880 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:44.880 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:44.880 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:44.880 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.880 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:44.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:44.881 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:44.881 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:44.881 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:44.882 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:44.882 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:44.883 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:44.883 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:44.883 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:44.883 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:44.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:44.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:44.884 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:44.885 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:44.885 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:44.885 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.655178_183.1", 2021-09-06T13:03:44.885 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:44.886 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:44.886 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:44.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.655178Z", 2021-09-06T13:03:44.886 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:44.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.887 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.887 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:44.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:44.887 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:44.888 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:44.888 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:44.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:44.888 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:44.889 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.655178_183.1 2021-09-06T13:03:49.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:49.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:49.891 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:49.967 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:49.967 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:50.070 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:50.070 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:50.071 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:50.072 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:50.073 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:50.074 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:50.074 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:50.074 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:50.074 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.075 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:50.075 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:50.075 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:50.075 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:50.075 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:50.076 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:50.076 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:50.076 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:50.076 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:50.077 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:50.077 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:50.077 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.077 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:50.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:50.078 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:50.078 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:50.078 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:50.078 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:50.079 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:50.079 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:50.079 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:50.079 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:50.079 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:50.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:50.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:50.080 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:50.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:50.081 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:50.081 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:03:50.081 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:50.081 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:50.082 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:50.082 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:03:50.082 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:50.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.083 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:50.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:50.083 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:50.083 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:50.084 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:50.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:50.084 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:50.085 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T13:03:55.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:55.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:03:55.088 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:03:55.289 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:03:55.290 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:03:55.397 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:03:55.397 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:03:55.397 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:03:55.398 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:03:55.398 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:03:55.398 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:03:55.399 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:55.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:55.399 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:03:55.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:55.400 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:03:55.400 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:55.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:55.400 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:03:55.401 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:55.401 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:03:55.401 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:55.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:03:55.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:55.402 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:55.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:55.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:03:55.403 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:55.403 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:55.403 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:03:55.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:55.404 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:55.404 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:55.404 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:03:55.404 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:55.405 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.405 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:55.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:55.405 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:03:55.406 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:55.406 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:55.406 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:03:55.406 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:55.407 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:55.407 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:55.407 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:03:55.408 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:55.408 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.408 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:55.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:03:55.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:03:55.409 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:03:55.409 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:03:55.409 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.655178_183.1", 2021-09-06T13:03:55.410 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:03:55.410 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:03:55.410 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:03:55.410 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.655178Z", 2021-09-06T13:03:55.411 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:03:55.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.411 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:03:55.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:03:55.412 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:03:55.412 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:03:55.412 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:03:55.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:03:55.413 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:03:55.414 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.655178_183.1 2021-09-06T13:04:00.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:00.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:00.417 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:00.503 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:00.503 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:00.996 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:01.002 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.003 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:01.004 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:01.005 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:01.006 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:04:01.007 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:04:01.008 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:01.009 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:01.010 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:01.010 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:01.012 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:04:06.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:06.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:06.014 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:06.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:06.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:06.581 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:06.581 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:06.581 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:06.582 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:06.582 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:06.582 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:06.582 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:06.583 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:06.584 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:06.585 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:06.586 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:06.587 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.588 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:06.589 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:06.590 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:06.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:06.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:06.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:04:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:06.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:06.594 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:06.594 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:04:06.595 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:06.595 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:06.596 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:06.597 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:06.597 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:06.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:06.598 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:06.599 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:04:11.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:11.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:11.603 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:11.700 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:11.700 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:12.120 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:12.121 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:12.122 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:12.123 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:12.124 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:12.126 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:12.127 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:12.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:12.129 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:12.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:12.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:12.132 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:04:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:12.133 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:04:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.134 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:12.135 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:12.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:12.136 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:12.137 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:04:17.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:17.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:17.141 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:17.226 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:17.226 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:17.442 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:17.442 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:17.443 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:17.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:17.445 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:17.446 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:17.447 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:17.448 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:17.450 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:17.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:17.452 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:04:17.453 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:17.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:17.455 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:17.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:17.456 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:17.457 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:04:22.458 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:22.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:22.460 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:22.562 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:22.563 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:22.841 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:22.842 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:22.842 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:22.843 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:22.843 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:22.843 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:22.843 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:22.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:22.844 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:22.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:22.844 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:22.844 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:22.845 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:22.845 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:22.845 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:22.845 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:22.845 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:22.846 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:22.846 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:22.846 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.846 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:22.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:22.847 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:22.847 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:22.847 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:22.847 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:22.848 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:22.848 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:22.848 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:22.848 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:22.849 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:22.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.849 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:22.849 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:22.849 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:22.850 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:22.850 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:22.850 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:22.850 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:22.851 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:22.851 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:22.851 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:22.851 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:22.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:22.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:22.852 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:22.852 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:22.853 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:22.853 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:04:22.853 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:22.853 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:22.853 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:22.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:04:22.854 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:22.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.854 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.855 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:22.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:22.855 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:22.855 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:22.855 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:22.856 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:22.856 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:22.857 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:04:27.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:27.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:27.860 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:27.957 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:27.957 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:28.392 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:28.392 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:28.392 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:28.393 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:28.393 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:28.393 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:28.393 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:28.393 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:28.394 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:28.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:28.394 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:28.394 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:28.394 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:28.395 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:28.395 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:28.395 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:28.395 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:28.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:28.396 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:28.396 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:28.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:28.397 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:28.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:28.397 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:28.397 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:28.398 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:28.398 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:28.398 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:28.398 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:28.399 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:28.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:28.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:28.399 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:28.400 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:28.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:28.400 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:28.400 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:28.400 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:28.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:28.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:28.402 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:28.402 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:28.402 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:04:28.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:28.403 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:28.403 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:28.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:04:28.403 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:28.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.404 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:28.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:28.405 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:28.405 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:28.405 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:28.405 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:28.406 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:28.407 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.542715_178.1 2021-09-06T13:04:33.408 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:33.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:33.410 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:33.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:33.517 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:33.845 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:33.846 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:33.846 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:33.846 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:33.846 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:33.846 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:33.847 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:33.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:33.847 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:33.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:33.848 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:33.848 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:33.848 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:33.848 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:33.848 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:33.849 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:33.849 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:33.849 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:33.849 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:33.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:33.850 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:33.850 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:33.850 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:33.850 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:33.851 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:33.851 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:33.851 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:33.851 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:33.852 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:33.852 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:33.852 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.852 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:33.852 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:33.853 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:33.853 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:33.853 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:33.853 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:33.854 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:33.854 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:33.854 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:33.854 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:33.854 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:33.855 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.855 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:33.855 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:33.855 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:33.855 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:33.856 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:33.856 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:04:33.856 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:33.856 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:33.856 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:04:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:33.859 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:33.859 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:04:38.861 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:38.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:38.862 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:38.953 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:38.953 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:39.412 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:39.413 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:39.413 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:39.413 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:39.413 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:39.414 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:39.414 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:39.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:39.414 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:39.414 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:39.415 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:39.415 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:39.415 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:39.415 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:39.416 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:39.416 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:39.416 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:39.416 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:39.416 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:39.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.417 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:39.417 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:39.417 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:39.417 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:39.418 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:39.418 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:39.418 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:39.418 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:39.419 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:39.419 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:39.419 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:39.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.419 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:39.420 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:39.420 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:39.420 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:39.420 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:04:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:04:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:39.426 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:39.428 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:04:44.429 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:44.430 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:44.430 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:44.543 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:44.543 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:45.014 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:45.015 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:45.016 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:45.016 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:45.016 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:45.016 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:45.017 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:45.018 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:45.019 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:45.020 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:45.021 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:45.022 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:45.023 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:45.024 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:45.025 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:04:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:45.026 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:04:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.027 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:45.028 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:45.029 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:45.029 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:45.030 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T13:04:50.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:50.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:50.033 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:50.139 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:50.139 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:50.537 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:50.537 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:50.537 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:50.537 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:50.538 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:50.538 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:50.538 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:50.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:50.539 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:50.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:50.539 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:50.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:50.540 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:50.540 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:50.540 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:50.540 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:50.541 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:50.541 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:50.541 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:50.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:50.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:50.542 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:50.542 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:50.542 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:50.543 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:50.543 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:50.543 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:50.543 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:50.543 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:50.544 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:50.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:50.544 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:50.544 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:50.545 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:50.545 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:50.545 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:50.545 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:50.545 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:50.546 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:50.546 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:50.546 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:50.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.547 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:50.547 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:50.547 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:50.547 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:50.547 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:50.548 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:04:50.548 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:50.548 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:50.548 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:50.548 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:04:50.549 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:50.549 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.549 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.549 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:50.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:50.550 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:50.550 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:50.550 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:50.550 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:50.551 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:50.552 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.542715_178.1 2021-09-06T13:04:55.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:55.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:04:55.555 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:04:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:04:55.750 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:04:55.971 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:04:55.971 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:04:55.971 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:04:55.972 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:04:55.972 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:04:55.972 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:04:55.972 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:55.972 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:55.973 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:04:55.973 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:55.973 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:04:55.973 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:55.974 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:55.974 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:04:55.974 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:55.974 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:04:55.975 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:55.975 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:04:55.975 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:55.975 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:55.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:55.976 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:04:55.976 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:55.976 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:55.977 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:04:55.977 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:55.977 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:55.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:55.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:04:55.978 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:55.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:55.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:55.979 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:04:55.979 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:55.979 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:55.979 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:04:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:04:55.980 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:55.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:55.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:04:55.981 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:04:55.982 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:04:55.982 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:04:55.982 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:04:55.982 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:04:55.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:04:55.983 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:04:55.983 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:04:55.983 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:04:55.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.984 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:04:55.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:04:55.984 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:04:55.985 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:04:55.985 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:04:55.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:04:55.985 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:04:55.986 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T13:05:00.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:00.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:00.990 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:01.090 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:01.091 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:01.446 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:01.447 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:01.447 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:01.447 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:01.447 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:01.448 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:01.448 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:01.448 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:01.448 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:01.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:01.449 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:01.449 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:01.449 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:01.450 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:01.450 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:01.450 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:01.450 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:01.450 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:01.451 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:01.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.451 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:01.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:01.452 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:01.452 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:01.452 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:01.452 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:01.453 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:01.453 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:01.453 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:01.453 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:01.454 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:01.454 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.454 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:01.454 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:01.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:01.455 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:01.455 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:01.455 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:01.455 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:01.455 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:01.456 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:01.456 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:01.456 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:01.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:01.457 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:01.457 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:01.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:01.457 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:01.458 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:05:01.458 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:01.458 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:01.458 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:01.459 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:05:01.459 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:01.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.459 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:01.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:01.460 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:01.460 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:01.460 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:01.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:01.461 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:01.462 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:05:06.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:06.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:06.464 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:06.622 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:06.622 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:06.902 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:06.902 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:06.903 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:06.903 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:06.903 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:06.903 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:06.903 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:06.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:06.904 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:06.904 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:06.904 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:06.905 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:06.905 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:06.905 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:06.905 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:06.905 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:06.906 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:06.906 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:06.906 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:06.906 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:06.907 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:06.908 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:06.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:06.910 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:06.911 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:06.911 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:06.911 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:06.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:06.912 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:06.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:06.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:06.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:06.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:06.913 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:06.913 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:05:06.913 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:06.914 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:06.914 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:06.914 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:05:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:06.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:06.916 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:06.917 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:06.918 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.718130_186.1 2021-09-06T13:05:11.919 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:11.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:11.921 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:12.041 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:12.042 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:12.455 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:12.456 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:12.457 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:12.458 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:12.459 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:12.460 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:12.461 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:12.462 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:12.463 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:12.464 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:12.464 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:12.464 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:12.465 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:12.466 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:12.467 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:05:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:12.468 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:05:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.469 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:12.470 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:12.471 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:12.471 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:12.473 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:05:17.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:17.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:17.475 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:17.556 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:17.557 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:18.025 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:18.025 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:18.026 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:18.026 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:18.026 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:18.026 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:18.027 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:18.027 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:18.027 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:18.027 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:05:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:05:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:18.040 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:18.041 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:05:23.043 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:23.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:23.044 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:23.143 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:23.143 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:23.408 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:23.409 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:23.409 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:23.409 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:23.409 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:23.409 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:23.410 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:23.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:23.410 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:23.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:23.411 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:23.411 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:23.411 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:23.411 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:23.411 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:23.412 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:23.412 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:23.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:23.412 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:23.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:23.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:23.413 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:23.413 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:23.414 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:23.414 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:23.414 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:23.414 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:23.414 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:23.415 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:23.415 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:23.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:23.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:23.416 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:23.417 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:23.417 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:23.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:23.417 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:23.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:23.418 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:23.418 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:23.418 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:23.419 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:23.419 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:05:23.419 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:23.419 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:23.420 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:23.420 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:05:23.420 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:23.420 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:23.421 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:23.422 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:23.422 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:23.422 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:23.423 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:05:28.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:28.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:28.425 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:28.507 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:28.507 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:28.678 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:28.679 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:28.679 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:28.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:28.680 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:28.680 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:28.680 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:28.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:28.681 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:28.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:28.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:28.682 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:28.682 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:28.682 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:28.682 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:28.683 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:28.683 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:28.683 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:28.683 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:28.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:28.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:28.684 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:28.684 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:28.685 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:28.685 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:28.685 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:28.685 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:28.685 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:28.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:28.686 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:28.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:28.687 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:28.688 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:28.688 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:28.688 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:28.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:28.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:28.689 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:28.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:28.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:28.690 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:28.690 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:28.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:05:28.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:28.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:28.691 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:28.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:05:28.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:28.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.692 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:28.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:28.692 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:28.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:28.693 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:28.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:28.693 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:28.694 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:05:33.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:33.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:33.697 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:33.819 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:33.820 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:34.192 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:34.192 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:34.193 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:34.194 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:34.195 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:34.196 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:34.198 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:34.199 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:34.201 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:34.202 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:34.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:34.204 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:05:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:34.205 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:05:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:34.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:34.207 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:34.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:34.208 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:34.210 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:05:39.211 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:39.212 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:39.213 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:39.311 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:39.311 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:39.740 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:39.744 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:39.744 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:39.745 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:39.745 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:39.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:39.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:39.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:39.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:39.749 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:39.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:39.750 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:39.751 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:39.752 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:39.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:39.755 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:39.757 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:05:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:39.758 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:05:39.759 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:39.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.760 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:39.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:39.760 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:39.761 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:39.762 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:39.762 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:39.763 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.471383_170.1 2021-09-06T13:05:44.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:44.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:44.766 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:44.842 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:44.842 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:45.198 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:45.199 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:45.199 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:45.199 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:45.199 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:45.200 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:45.200 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:45.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:45.200 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:45.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:45.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:45.201 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:45.201 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:45.202 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:45.202 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:45.202 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:45.202 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:45.203 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:45.203 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:45.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.203 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:45.203 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:45.204 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:45.204 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:45.204 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:45.204 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:45.204 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:45.205 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:45.205 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:45.205 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:45.205 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:45.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:45.206 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:45.206 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:45.206 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:45.207 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:45.207 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:45.207 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:45.207 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:45.208 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:45.208 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:45.208 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:45.208 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.208 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:45.209 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:45.209 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:45.209 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:45.209 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:45.210 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:05:45.210 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:45.210 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:45.210 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:45.210 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:05:45.211 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:45.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.211 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.211 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:45.212 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:45.212 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:45.212 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:45.212 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:45.212 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:45.213 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:45.214 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.474951_171.1 2021-09-06T13:05:50.215 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:50.216 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:50.217 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:50.318 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:50.318 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:50.468 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:50.469 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:50.469 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:50.469 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:50.469 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:50.469 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:50.470 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:50.470 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:50.470 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:50.470 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:50.471 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:50.471 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:50.471 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:50.471 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:50.472 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:50.472 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:50.472 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:50.472 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:50.473 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:50.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.473 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:50.473 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:50.473 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:50.474 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:50.474 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:50.474 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:50.474 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:50.475 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:50.475 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:50.475 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:50.475 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:50.476 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:50.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:50.476 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:50.476 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:50.477 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:50.477 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:50.477 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:50.477 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:50.477 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:50.478 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:50.478 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:50.478 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.478 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:50.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:50.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:50.479 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:50.479 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:50.480 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:05:50.480 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:50.480 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:50.480 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:50.480 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:05:50.481 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:50.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.481 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:50.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:50.482 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:50.482 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:50.482 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:50.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:50.483 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:50.484 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:05:55.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:55.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:05:55.487 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:05:55.577 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:05:55.577 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:05:56.107 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:05:56.107 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:05:56.108 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:05:56.108 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:05:56.108 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:05:56.108 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:05:56.109 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:56.109 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:56.109 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:05:56.109 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:56.110 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:05:56.110 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:56.110 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:56.110 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:05:56.110 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:56.111 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:05:56.111 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:56.111 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:05:56.111 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:56.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:56.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:56.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:05:56.112 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:56.113 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:56.113 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:05:56.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:56.113 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:56.113 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:56.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:05:56.114 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:56.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.114 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:56.114 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:56.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:05:56.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:56.115 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:56.115 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:05:56.115 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:05:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:56.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:56.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:05:56.117 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:05:56.117 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:05:56.117 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:05:56.118 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:05:56.118 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:05:56.118 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:05:56.118 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:05:56.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:05:56.119 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:05:56.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:05:56.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:05:56.120 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:05:56.120 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:05:56.120 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:05:56.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:05:56.121 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:05:56.122 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:06:01.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:01.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:01.124 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:01.212 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:01.213 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:01.521 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:01.521 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:01.521 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:01.522 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:01.522 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:01.522 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:01.522 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:01.523 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:01.523 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:01.523 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:01.523 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:01.523 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:01.524 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:01.524 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:01.524 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:01.525 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:01.525 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:01.525 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:01.525 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:01.525 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.526 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:01.526 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:01.526 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:01.526 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:01.526 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:01.527 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:01.527 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:01.527 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:01.528 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:01.528 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:01.528 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:01.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:01.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:01.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:01.529 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:01.529 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:01.529 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:01.530 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:01.530 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:01.530 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:01.530 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:01.531 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:01.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.531 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:01.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:01.532 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:01.532 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:01.532 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:01.532 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:06:01.532 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:01.533 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:01.533 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:01.533 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:06:01.533 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:01.534 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.534 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.534 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:01.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:01.535 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:01.535 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:01.535 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:01.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:01.536 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:01.537 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:06:06.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:06.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:06.540 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:06.617 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:06.618 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:07.123 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:07.123 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:07.123 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:07.124 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:07.125 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:07.126 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:07.126 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:07.126 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:07.126 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:07.126 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:07.127 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:07.127 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:07.127 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.127 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:07.128 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:07.128 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:07.128 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:07.128 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:07.128 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:07.129 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:07.129 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:07.129 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:07.129 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:07.130 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:07.130 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.130 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:07.130 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:07.131 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:07.131 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:07.131 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:07.131 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:07.131 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:07.132 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:07.132 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:07.132 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:07.132 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:07.133 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:07.133 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:07.133 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:07.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:07.134 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:07.134 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:06:07.134 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:07.134 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:07.135 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:07.135 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:06:07.135 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:07.135 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.136 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:07.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:07.136 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:07.136 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:07.137 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:07.137 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:07.137 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:07.138 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:06:12.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:12.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:12.141 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:12.184 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:12.184 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:12.687 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:12.687 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:12.687 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:12.688 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:12.688 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:12.688 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:12.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:12.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:12.689 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:12.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:12.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:12.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:12.690 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:12.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:12.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:12.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:12.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:12.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:12.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:12.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:12.692 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:12.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:12.692 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:12.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:12.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:12.693 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:12.693 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:12.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:12.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:12.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:12.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:12.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:12.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:12.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:12.695 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:12.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:12.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:12.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:12.696 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:12.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:12.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:12.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:12.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:12.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:12.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:12.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:12.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:06:12.698 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:12.698 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:12.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:12.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:06:12.699 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:12.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.699 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:12.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:12.700 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:12.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:12.700 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:12.700 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:12.700 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:12.702 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:06:17.703 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:17.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:17.705 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:17.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:17.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:18.074 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:18.075 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:18.075 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:18.075 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:18.076 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:18.076 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:18.076 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:18.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:18.077 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:18.077 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:18.077 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:18.077 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:18.078 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:18.078 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:18.078 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:18.078 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:18.079 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:18.079 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:18.079 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:18.079 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:18.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:18.080 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:18.080 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:18.080 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:18.081 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:18.081 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:18.081 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:18.081 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:18.082 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:18.082 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:18.082 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.082 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:18.082 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:18.083 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:18.083 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:18.083 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:18.083 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:18.084 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:18.084 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:18.084 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:18.084 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:18.085 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:18.085 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.085 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:18.085 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:18.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:18.086 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:18.086 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:18.086 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:06:18.086 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:18.087 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:18.087 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:18.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:06:18.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:18.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.088 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.088 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:18.088 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:18.088 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:18.088 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:18.089 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:18.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:18.089 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:18.090 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:06:23.091 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:23.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:23.094 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:23.201 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:23.201 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:23.589 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:23.590 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:23.590 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:23.590 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:23.590 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:23.591 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:23.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:23.591 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:23.591 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:23.591 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:23.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:23.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:23.592 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:23.592 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:23.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:23.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:23.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:23.593 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:23.593 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:23.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:23.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:23.594 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:23.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:23.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:23.595 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:23.595 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:23.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:23.596 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:23.596 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:23.596 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:23.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:23.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:23.597 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:23.598 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:23.598 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:23.598 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:23.598 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:23.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:23.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:23.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:23.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:23.601 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:23.601 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:06:23.601 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:23.601 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:23.602 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:23.602 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:06:23.602 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:23.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.603 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:23.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:23.603 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:23.603 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:23.604 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:23.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:23.604 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:23.605 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:06:28.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:28.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:28.608 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:28.704 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:28.705 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:29.092 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:29.092 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:29.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:29.094 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:29.095 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:29.096 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:29.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:29.098 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:29.099 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:29.100 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:29.101 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:29.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:29.103 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:29.104 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:29.105 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:29.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.106 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:29.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:29.107 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:29.107 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:29.108 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:29.108 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:29.109 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:29.109 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:29.110 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:29.110 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:29.111 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:29.111 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.111 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:29.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:29.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:29.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:29.113 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:29.114 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:06:29.114 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:29.114 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:29.115 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:29.115 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:06:29.116 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:29.116 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.117 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:29.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:29.118 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:29.118 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:29.119 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:29.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:29.120 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:29.122 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:06:34.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:34.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:34.126 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:34.221 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:34.222 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:34.678 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:34.679 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:34.679 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:34.679 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:06:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:06:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:34.693 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:34.694 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.454323_168.1 2021-09-06T13:06:39.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:39.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:39.696 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:39.792 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:40.240 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:40.240 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:40.240 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:40.241 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:40.241 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:40.241 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:40.241 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:40.242 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:40.242 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:40.242 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:40.242 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:40.243 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:40.243 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:40.243 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:40.243 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:40.244 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:40.244 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:40.244 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:40.244 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:40.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.245 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:40.245 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:40.245 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:40.245 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:40.246 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:40.246 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:40.246 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:40.246 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:40.247 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:40.247 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:40.247 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:40.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.248 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:40.248 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:40.248 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:40.248 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:40.248 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:40.249 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:40.249 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:40.249 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:40.250 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:40.250 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:40.250 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:06:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:06:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:40.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:40.256 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:40.256 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.510450_174.1 2021-09-06T13:06:45.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:45.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:45.259 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:45.361 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:45.364 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:45.554 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:45.555 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:45.555 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:45.555 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:45.555 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:45.556 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:45.556 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:45.556 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:45.556 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:45.557 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:45.557 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:45.557 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:45.557 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:45.557 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:45.558 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:45.558 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:45.558 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:45.558 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:45.559 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:45.559 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:45.559 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:45.560 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:45.560 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:45.560 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:45.560 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:45.561 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:45.561 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:45.561 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:45.561 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:45.562 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:45.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:45.562 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:45.563 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:45.564 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:45.564 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:45.564 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:45.565 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:45.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:45.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:45.565 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:45.566 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:45.566 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:45.566 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:06:45.566 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:45.567 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:45.567 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:45.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:06:45.567 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:45.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.568 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:45.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:45.569 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:45.569 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:45.569 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:45.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:45.570 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:45.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.476870_172.1 2021-09-06T13:06:50.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:50.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:50.574 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:50.673 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:50.673 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:50.780 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:50.781 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:50.781 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:50.781 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:50.781 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:50.782 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:50.782 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:50.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:50.782 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:50.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:50.783 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:50.783 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:50.783 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:50.784 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:50.784 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:50.784 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:50.785 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:50.785 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:50.785 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:50.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.786 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:50.786 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:50.786 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:50.786 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:50.787 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:50.787 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:50.787 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:50.787 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:50.788 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:50.788 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:50.788 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:50.789 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.789 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:50.789 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:50.789 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:50.790 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:50.790 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:50.790 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:50.790 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:50.791 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:50.791 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:50.791 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:50.791 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:50.792 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:50.792 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:50.792 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:50.792 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:50.793 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:50.793 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:06:50.793 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:50.793 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:50.793 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:50.794 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:06:50.794 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:50.794 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.794 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.795 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:50.795 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:50.795 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:50.795 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:50.796 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:50.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:50.796 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:50.797 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.453043_167.1 2021-09-06T13:06:55.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:55.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:06:55.800 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:06:55.885 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:06:55.886 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:06:56.163 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:06:56.164 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:06:56.164 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:06:56.164 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:06:56.165 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:06:56.165 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:06:56.165 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:56.165 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:56.165 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:06:56.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:56.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:06:56.166 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:56.166 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:56.167 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:06:56.167 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:56.167 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:06:56.167 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:56.167 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:06:56.168 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:56.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:56.168 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:56.168 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:06:56.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:56.169 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:56.169 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:06:56.169 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:56.169 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:56.170 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:56.170 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:06:56.170 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:56.170 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.171 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:56.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:56.171 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:06:56.171 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:56.172 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:56.172 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:06:56.172 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:56.172 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:56.173 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:56.173 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:06:56.173 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:56.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:56.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:06:56.174 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:06:56.174 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:06:56.174 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:06:56.174 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:06:56.175 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:06:56.175 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:06:56.175 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:06:56.175 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:06:56.176 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:06:56.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.176 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:06:56.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:06:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:06:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:06:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:06:56.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:06:56.177 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:06:56.179 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.457681_169.1 2021-09-06T13:07:01.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:01.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:01.181 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:01.277 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:01.278 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:01.617 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:01.618 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:01.618 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:01.618 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:01.618 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:01.619 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:01.620 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:01.620 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:01.620 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:01.620 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:01.621 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:01.621 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:01.621 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:01.621 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:01.621 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:01.622 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.622 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:01.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:01.622 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:01.623 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:01.623 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:01.623 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:01.623 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:01.623 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:01.624 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:01.624 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:01.624 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:01.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:01.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:01.625 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:01.625 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:01.625 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:01.626 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:01.626 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:01.626 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:01.626 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:01.627 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:01.627 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:01.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:01.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:01.628 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:01.628 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:01.628 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:01.628 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.742338_191.1", 2021-09-06T13:07:01.628 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:01.629 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:01.629 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:01.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.742338Z", 2021-09-06T13:07:01.629 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:01.629 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.630 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:01.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:01.630 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:01.631 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:01.631 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:01.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:01.631 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:01.632 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.742338_191.1 2021-09-06T13:07:06.633 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:06.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:06.635 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:06.740 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:06.741 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:06.931 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:06.931 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:06.931 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:06.931 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:06.932 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:06.933 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:06.934 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:06.935 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:06.936 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:06.938 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:06.939 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:06.940 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:07:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:06.941 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:07:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.943 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:06.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:06.943 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:06.943 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:06.943 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:06.944 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:06.944 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:06.945 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933021.264676_258.1 target=1_1630932661.483094_173.1 2021-09-06T13:07:11.947 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... FAIL 2021-09-06T13:07:11.948 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-42 2021-09-06T13:07:11.985 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-43 2021-09-06T13:07:13.166 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-44 2021-09-06T13:07:13.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:13.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:13.352 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:13.397 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:13.398 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:13.416 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T13:07:13.417+0000 7f8cccc3a700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T13:07:13.449 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T13:07:13.466 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T13:07:13.466 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:13.467 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:13.467 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T13:07:13.467 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:13.467 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:13.468 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:07:13.468 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T13:07:13.468 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:13.468 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:13.469 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:13.469 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T13:07:13.469 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:13.469 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:13.469 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:07:13.470 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T13:07:13.195130Z" 2021-09-06T13:07:13.470 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:13.470 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T13:07:13.471 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T13:07:13.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:13.472 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:13.473 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:13.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:13.517 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:13.594 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:13.594 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:13.594 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:13.594 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:13.595 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:13.595 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:13.595 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:13.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:13.596 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:13.596 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:13.596 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:13.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:13.597 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:13.597 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:13.597 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:13.597 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:13.598 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:13.598 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:13.598 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:13.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.599 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:13.599 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:13.599 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:13.599 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:13.599 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:13.600 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:13.600 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:13.600 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:13.600 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:13.601 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:13.601 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:13.601 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.601 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:13.601 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:13.602 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:13.602 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:13.602 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:13.602 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:13.602 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:13.603 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:13.603 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:13.603 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:13.603 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:13.604 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:13.605 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.086252_271.1", 2021-09-06T13:07:13.605 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:13.605 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:13.605 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:13.606 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.086252Z", 2021-09-06T13:07:13.606 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:13.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.606 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:13.607 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:13.607 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:13.607 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:13.607 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:13.607 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:13.608 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:13.609 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630933633.086252_271.1 2021-09-06T13:07:14.620 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T13:07:14.619+0000 7f8ccb437700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T13:07:14.621 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T13:07:14.620+0000 7f8cc742f700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T13:07:14.658 INFO:tasks.rgw.c2.client.1.smithi052.stdout:2021-09-06T13:07:14.659+0000 7f8cc642d700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T13:07:18.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.611 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.611 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:18.656 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:18.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:18.722 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:18.722 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:18.723 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:18.723 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:18.723 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:18.723 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:18.724 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.724 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.724 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:18.724 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:18.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.725 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.725 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:18.725 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.726 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:18.726 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.726 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:18.726 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.726 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.727 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.727 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:18.727 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.728 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.728 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:18.728 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.728 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:18.728 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.729 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:18.729 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.729 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.730 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:18.730 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.730 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.730 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:18.731 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.731 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:18.731 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.731 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:18.732 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.732 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:18.733 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.733 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.733 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:07:18.733 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.734 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:18.734 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.734 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:07:18.734 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.735 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:18.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.735 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:18.735 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:18.736 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:18.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.736 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:18.737 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T13:07:18.738 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T13:07:18.738 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.739 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:18.789 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:18.790 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:18.862 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:07:18.862 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:07:18.863 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:07:18.863 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:07:18.863 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:07:18.863 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:18.864 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:07:18.864 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:18.864 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:07:18.864 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:18.864 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:07:18.865 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:07:18.865 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:07:18.865 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:18.865 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:07:18.865 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:07:18.866 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:07:18.866 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:18.866 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:07:18.866 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.866 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:18.867 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:18.867 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:07:18.867 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:07:18.867 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:07:18.868 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:07:18.868 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:07:18.868 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:07:18.868 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:07:18.868 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:07:18.869 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:07:18.869 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.869 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:18.869 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:18.869 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:07:18.870 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:07:18.870 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:07:18.870 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:18.870 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:07:18.870 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:07:18.871 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:07:18.871 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:18.871 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:07:18.871 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.871 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:18.872 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:07:18.872 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:07:18.872 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:07:18.872 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:07:18.872 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:07:18.873 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:07:18.873 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:07:18.873 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:07:18.873 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:07:18.874 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:07:18.874 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.874 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.874 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:07:18.874 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:07:18.875 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:07:18.875 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:07:18.875 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:07:18.875 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:07:18.876 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:07:18.877 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T13:07:18.877 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T13:07:18.877 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:18.879 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:18.923 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:18.923 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:18.989 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:18.989 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:18.989 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:18.989 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.990 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.991 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.992 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.993 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:18.994 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:18.996 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:18.997 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:18.998 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:18.999 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:07:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:07:19.000 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:19.001 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:19.002 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:19.002 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:19.003 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.517790_176.1 2021-09-06T13:07:24.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:24.005 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:24.006 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:24.070 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:24.071 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:24.135 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:24.135 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:24.135 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:24.136 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:24.136 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:24.136 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:24.136 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:24.137 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:24.138 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:24.140 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:24.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:24.142 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:24.143 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:24.144 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:24.145 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:07:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:24.146 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:07:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.147 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:24.148 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:24.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:24.149 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:24.150 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.510450_174.1 2021-09-06T13:07:29.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:29.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:29.152 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:29.196 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:29.196 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:29.261 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:29.261 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:29.262 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:29.262 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:29.262 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:29.262 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:29.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:29.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:29.263 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:29.263 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:29.263 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:29.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:29.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:29.264 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:29.264 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:29.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:29.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:29.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:29.265 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:29.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:29.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:29.267 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:29.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:29.267 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:29.267 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:29.267 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:29.268 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:29.268 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:29.268 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:29.268 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:29.268 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:29.269 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:29.269 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:29.269 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:29.270 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:29.270 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:29.270 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:29.270 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:29.271 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:29.271 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:29.271 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:29.271 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.271 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:29.272 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:29.272 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:29.272 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:29.272 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:29.273 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:07:29.273 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:29.273 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:29.273 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:29.273 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:07:29.274 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:29.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.274 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.274 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:29.275 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:29.275 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:29.275 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:29.275 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:29.275 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:29.276 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:29.277 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:07:34.278 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:34.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:34.280 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:34.332 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:34.332 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:34.405 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:34.405 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:34.406 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:34.406 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:34.406 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:34.407 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:34.407 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:34.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:34.407 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:34.408 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:34.408 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:34.408 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:34.408 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:34.408 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:34.409 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:34.409 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:34.409 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:34.409 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:34.410 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:34.410 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:34.410 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:34.410 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:34.411 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:34.411 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:34.411 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:34.411 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:34.412 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:34.412 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:34.412 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:34.412 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:34.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.413 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:34.413 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:34.413 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:34.413 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:34.414 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:34.414 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:34.414 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:34.414 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:34.414 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:34.415 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:34.415 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:34.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.415 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:34.415 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:34.416 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:34.416 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:34.416 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:34.416 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T13:07:34.416 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T13:07:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:34.417 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:34.418 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:34.419 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:34.419 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:34.420 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.713479_184.1 2021-09-06T13:07:39.422 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:39.422 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:39.423 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:39.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:39.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:39.585 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:39.585 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:39.585 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:39.585 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:39.586 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:39.586 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:39.586 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:39.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:39.587 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:39.587 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:39.587 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:39.587 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:39.587 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:39.588 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:39.588 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:39.588 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:39.588 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:39.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:39.589 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:39.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:39.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:39.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:39.590 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:39.590 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:39.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:39.590 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:39.591 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:39.591 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:39.591 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:39.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:39.591 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:39.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:39.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:39.592 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:39.593 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:39.593 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:39.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:39.593 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:39.593 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:39.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:39.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:39.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:39.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:39.595 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:07:39.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:39.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:39.596 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:39.596 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:07:39.596 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:39.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.597 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:39.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:39.598 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:39.598 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:39.598 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:39.598 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:39.598 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:39.600 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:07:44.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:44.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:44.603 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:44.668 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:44.669 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:44.734 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:44.735 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:44.735 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:44.735 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:44.735 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:44.736 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:44.736 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:44.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:44.736 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:44.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:44.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:44.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:44.737 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:44.737 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:44.738 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:44.738 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:44.738 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:44.738 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:44.739 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:44.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:44.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:44.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:44.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:44.740 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:44.740 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:44.740 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:44.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:44.741 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:44.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:44.741 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:44.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:44.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:44.742 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:44.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:44.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:44.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:44.743 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:44.743 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:44.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:44.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:44.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:44.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:44.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:44.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:44.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:44.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:44.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:07:44.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:44.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:44.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:44.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:07:44.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:44.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.747 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:44.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:44.749 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:44.750 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:07:49.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:49.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:49.753 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:49.844 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:49.844 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:50.379 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:50.379 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:50.380 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:50.380 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:50.380 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:50.380 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:50.381 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:50.381 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:50.381 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:50.381 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:50.381 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:50.382 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:50.382 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:50.382 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:50.382 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:50.383 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:50.383 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:50.383 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:50.383 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:50.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.384 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:50.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:50.384 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:50.384 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:50.385 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:50.385 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:50.385 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:50.385 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:50.386 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:50.386 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:50.386 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:50.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:50.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:50.387 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:50.387 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:50.387 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:50.388 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:50.388 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:50.388 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:50.388 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:50.388 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:50.389 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:50.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:50.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:50.390 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:50.390 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:50.390 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:50.390 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:07:50.390 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:50.391 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:50.391 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:50.391 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:07:50.391 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:50.391 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.392 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.392 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:50.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:50.392 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:50.393 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:50.393 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:50.393 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:50.393 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:50.394 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:07:55.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:55.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:07:55.397 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:07:55.460 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:07:55.460 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:07:55.809 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:07:55.809 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:07:55.810 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:07:55.810 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:07:55.810 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:07:55.811 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:07:55.811 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:55.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:55.811 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:07:55.811 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:55.812 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:07:55.812 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:55.812 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:55.812 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:07:55.813 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:55.813 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:07:55.813 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:55.813 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:07:55.814 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:55.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:55.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:55.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:07:55.815 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:55.815 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:55.815 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:07:55.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:55.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:55.816 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:55.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:07:55.816 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:55.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:55.817 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:55.817 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:07:55.817 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:55.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:55.818 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:07:55.818 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:55.818 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:55.818 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:55.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:07:55.819 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:55.819 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.819 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:55.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:07:55.820 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:07:55.820 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:07:55.820 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:07:55.820 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:07:55.820 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:07:55.821 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:07:55.821 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:07:55.821 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:07:55.821 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:07:55.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.822 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:07:55.822 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:07:55.822 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:07:55.823 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:07:55.823 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:07:55.823 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:07:55.823 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:07:55.825 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.516029_175.1 2021-09-06T13:08:00.826 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:00.827 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:00.827 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:00.913 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:00.913 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:01.033 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:01.033 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:01.034 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:01.035 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:01.036 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:01.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:01.038 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:01.039 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:01.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:01.041 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:01.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:01.043 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:08:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:01.044 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:08:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:01.046 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:01.047 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:01.047 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:01.048 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:08:06.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:06.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:06.051 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:06.100 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:06.100 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:06.172 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:06.173 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:06.174 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:06.175 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:06.177 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:06.178 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:06.179 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:06.180 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:06.181 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:06.182 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:06.182 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:06.182 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:06.182 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:06.183 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:08:06.184 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:06.184 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:06.184 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:06.184 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:08:06.184 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:06.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.185 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:06.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:06.186 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:06.186 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:06.186 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:06.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:06.186 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:06.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.483094_173.1 2021-09-06T13:08:11.189 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:11.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:11.190 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:11.235 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:11.235 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:11.301 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:11.301 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:11.301 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:11.302 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:11.302 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:11.302 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:11.302 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:11.303 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:11.303 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:11.303 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:11.303 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:11.303 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:11.304 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:11.304 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:11.304 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:11.304 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:11.305 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:11.305 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:11.305 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:11.305 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.305 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:11.306 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:11.306 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:11.306 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:11.306 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:11.306 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:11.307 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:11.307 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:11.307 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:11.307 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:11.308 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:11.309 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:11.309 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:11.309 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:11.309 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:11.310 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:11.310 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:11.310 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:11.310 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:11.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:11.311 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:11.311 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:11.311 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:11.311 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:08:11.312 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:11.312 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:11.312 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:11.312 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:08:11.312 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:11.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.313 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:11.313 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:11.313 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:11.314 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:11.314 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:11.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:11.314 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:11.315 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.650194_182.1 2021-09-06T13:08:16.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:16.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:16.318 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:16.406 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:16.406 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:16.515 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:16.515 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:16.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:16.517 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:16.518 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:16.518 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:16.518 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:16.518 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:16.518 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:16.519 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:16.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:16.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:16.519 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:16.520 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:16.520 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:16.520 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:16.520 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:16.520 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:16.521 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:16.521 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:16.521 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:16.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:16.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:16.522 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:16.522 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:16.522 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:16.522 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:16.523 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:16.523 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:16.523 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:16.523 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:16.523 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:16.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:16.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:16.524 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:16.524 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:08:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:16.525 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:16.526 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:08:16.526 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:16.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.526 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:16.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:16.527 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:16.527 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:16.527 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:16.527 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:16.528 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:16.529 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.595266_180.1 2021-09-06T13:08:21.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:21.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:21.532 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:21.575 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:21.575 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:21.878 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:21.878 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:21.879 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:21.880 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:21.881 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.882 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:21.883 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:21.884 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.885 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:21.886 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:21.887 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:21.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:21.889 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:08:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:08:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:21.893 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:21.894 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.474951_171.1 2021-09-06T13:08:26.895 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:26.896 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:26.896 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:26.981 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:26.981 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:27.307 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:27.307 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:27.307 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:27.308 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:27.308 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:27.308 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:27.309 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:27.309 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:27.309 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:27.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:27.310 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:27.311 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:27.311 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:27.311 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:27.312 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:27.312 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:27.312 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:27.313 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:27.313 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:27.313 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:27.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:27.314 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:27.314 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:27.315 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:27.315 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:27.315 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:27.315 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:27.315 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:27.316 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:27.316 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:27.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:27.316 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:27.317 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:27.317 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:27.317 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:27.317 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:27.318 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:27.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:27.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:27.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:27.319 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:27.320 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:27.320 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.737368_189.1", 2021-09-06T13:08:27.320 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:27.320 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:27.321 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:27.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.737368Z", 2021-09-06T13:08:27.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:27.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.322 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:27.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:27.322 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:27.322 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:27.322 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:27.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:27.323 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:27.324 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.737368_189.1 2021-09-06T13:08:32.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:32.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:32.326 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:32.387 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:32.952 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:32.952 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:32.953 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:32.953 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:32.953 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:32.953 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:32.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:32.954 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:32.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:32.954 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:32.954 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:32.955 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:32.955 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:32.955 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:32.955 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:32.956 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:32.956 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:32.956 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:32.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:32.957 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:32.957 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:32.957 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:32.957 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:32.957 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:32.958 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:32.958 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:32.958 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:32.958 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:32.959 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:32.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:32.959 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:32.959 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:32.960 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:32.960 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:32.960 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:32.960 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:32.960 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:32.961 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:32.961 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:32.961 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:32.961 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.962 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:32.962 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:32.962 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:32.962 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:32.962 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:08:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:32.963 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:08:32.964 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:32.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.964 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.964 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:32.964 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:32.965 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:32.965 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:32.965 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:32.965 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:32.965 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:32.967 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:08:37.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:37.969 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:37.970 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:38.060 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:38.060 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:38.650 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:38.651 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:38.651 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:38.651 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:38.651 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:38.652 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:38.652 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:38.652 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:38.652 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:38.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:38.653 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:38.653 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:38.653 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:38.654 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:38.654 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:38.654 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:38.654 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:38.655 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:38.655 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:38.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.655 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:38.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:38.656 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:38.656 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:38.656 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:38.656 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:38.657 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:38.657 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:38.657 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:38.657 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:38.657 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:38.658 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:38.658 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:38.658 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:38.658 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:38.659 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:38.659 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:38.659 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:38.659 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:38.660 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:38.660 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:38.660 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:38.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:38.661 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:38.662 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:08:38.662 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:38.662 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:38.662 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:38.663 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:08:38.663 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:38.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.663 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:38.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:38.664 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:38.664 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:38.664 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:38.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:38.665 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:38.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:08:43.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:43.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:43.669 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:43.748 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:43.748 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:44.402 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:44.402 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:44.402 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:44.402 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:44.403 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:44.403 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:44.403 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:44.403 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:44.404 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:44.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:44.404 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:44.404 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:44.404 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:44.405 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:44.405 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:44.405 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:44.405 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:44.406 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:44.406 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:44.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.406 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:44.406 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:44.407 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:44.407 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:44.407 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:44.407 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:44.408 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:44.408 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:44.408 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:44.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:44.408 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:44.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:44.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:44.409 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:44.409 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:44.410 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:44.410 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:44.410 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:44.410 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:44.411 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:44.411 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:44.411 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:44.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.411 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:44.412 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:44.412 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:44.412 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:44.412 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:44.413 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:08:44.413 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:44.413 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:44.413 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:44.413 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:08:44.414 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:44.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.414 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.414 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:44.414 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:44.415 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:44.415 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:44.415 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:44.415 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:44.415 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:44.416 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.474951_171.1 2021-09-06T13:08:49.417 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:49.418 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:49.419 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:49.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:49.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:49.979 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:49.980 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:49.980 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:49.980 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:49.980 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:49.981 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:49.981 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:49.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:49.981 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:49.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:49.982 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:49.982 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:49.982 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:49.983 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:49.983 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:49.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:49.983 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:49.984 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:49.984 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:49.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:49.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:49.985 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:49.985 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:49.985 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:49.985 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:49.986 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:49.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:49.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:49.987 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:49.988 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:49.988 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:49.988 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:49.988 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:49.988 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:49.989 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:49.989 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:49.989 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:49.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.990 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:49.990 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:49.990 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:49.990 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:49.991 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:49.991 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.105685_211.1", 2021-09-06T13:08:49.991 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:49.991 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:49.991 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:49.992 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.105685Z", 2021-09-06T13:08:49.992 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:49.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.993 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:49.993 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:49.993 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:49.993 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:49.993 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:49.994 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:49.994 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:49.995 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932662.105685_211.1 2021-09-06T13:08:54.997 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:54.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:08:54.999 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:08:55.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:08:55.082 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:08:55.651 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:08:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:08:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:08:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:08:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:08:55.652 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:08:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:08:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:55.653 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:08:55.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:55.654 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:55.654 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:08:55.654 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:55.654 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:08:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:08:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:55.655 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:08:55.656 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:55.657 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:08:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.658 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:08:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:55.659 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:08:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:08:55.660 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:08:55.661 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:08:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:08:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:08:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:08:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:08:55.662 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:08:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:08:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:08:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:08:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.663 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:08:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:08:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:08:55.664 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:08:55.665 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:08:55.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:08:55.665 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:08:55.666 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:09:00.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:00.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:00.669 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:00.719 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:00.719 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:00.825 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:00.825 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:00.825 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:00.825 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:00.826 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:00.826 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:00.826 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:00.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:00.827 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:00.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:00.827 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:00.827 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:00.828 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:00.829 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:00.829 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:00.829 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:00.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:00.830 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:00.830 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:00.830 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:00.831 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:00.831 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:00.831 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:00.831 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:00.832 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:00.833 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:00.833 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:00.833 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:00.833 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:00.833 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:00.834 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:00.834 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:00.834 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:00.834 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:00.835 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:00.835 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:00.835 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:00.835 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:00.836 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:00.836 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.742338_191.1", 2021-09-06T13:09:00.836 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:00.836 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:00.836 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:00.837 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.742338Z", 2021-09-06T13:09:00.837 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:00.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.837 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.838 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:00.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:00.838 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:00.838 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:00.838 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:00.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:00.839 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:00.840 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.742338_191.1 2021-09-06T13:09:05.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:05.842 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:05.843 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:06.071 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:06.071 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:06.144 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:06.145 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:06.146 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:06.147 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:06.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:06.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:06.150 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:06.151 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:06.152 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:06.153 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:06.154 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:06.155 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.125517_212.1", 2021-09-06T13:09:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:06.156 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.125517Z", 2021-09-06T13:09:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.157 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:06.158 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:06.159 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:06.159 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:06.160 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932662.125517_212.1 2021-09-06T13:09:11.161 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:11.162 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:11.163 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:11.207 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:11.207 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:11.271 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:11.272 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:11.272 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:11.272 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:11.272 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:11.273 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:11.273 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:11.273 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:11.273 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:11.274 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:11.274 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:11.274 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:11.274 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:11.274 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:11.275 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:11.275 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:11.275 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:11.275 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:11.276 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:11.276 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.276 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:11.276 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:11.276 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:11.277 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:11.277 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:11.277 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:11.277 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:11.278 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:11.278 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:11.281 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:11.281 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:11.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:11.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:11.283 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:11.283 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:11.283 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:11.283 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:11.284 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:11.284 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:11.284 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:11.284 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:11.284 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:11.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:11.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:11.285 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:11.286 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:11.286 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:11.286 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:09:11.286 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:11.286 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:11.287 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:11.287 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:09:11.287 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:11.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.288 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:11.288 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:11.288 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:11.288 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:11.288 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:11.289 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:11.289 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:11.290 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.457681_169.1 2021-09-06T13:09:16.291 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:16.292 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:16.293 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:16.357 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:16.357 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:16.742 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:16.743 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:16.743 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:16.743 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:16.744 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:16.744 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:16.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:16.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:16.744 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:16.745 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:16.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:16.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:16.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:16.746 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:16.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:16.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:16.747 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:16.747 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:16.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:16.748 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:16.748 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:16.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:16.749 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:16.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:16.749 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:16.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:16.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:16.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:16.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:16.750 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:16.751 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:16.752 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:16.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.752 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:16.752 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:16.752 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:16.753 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:16.753 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:16.753 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:09:16.753 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:16.754 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:16.754 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:16.754 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:09:16.754 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:16.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.755 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:16.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:16.755 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:16.756 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:16.756 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:16.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:16.756 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:16.757 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.510450_174.1 2021-09-06T13:09:21.758 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:21.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:21.761 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:21.822 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:21.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:21.890 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:21.890 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:21.891 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:21.892 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:21.893 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:21.894 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:21.895 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:21.896 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:21.896 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:21.896 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:21.898 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:21.899 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:21.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:21.901 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:09:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:21.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:09:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.903 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:21.904 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:21.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:21.905 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:21.906 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:09:26.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:26.908 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:26.909 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:27.000 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:27.001 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:27.514 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:27.515 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:27.516 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:27.517 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:27.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:27.519 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:27.520 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:27.522 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:27.523 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:27.524 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:09:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:27.525 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:09:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.526 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:27.527 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:27.528 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:27.528 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:27.529 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:09:32.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:32.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:32.531 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:32.581 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:32.582 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:32.700 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:32.701 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:32.701 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:32.701 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:32.701 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:32.702 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:32.702 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:32.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:32.702 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:32.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:32.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:32.703 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:32.703 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:32.703 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:32.704 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:32.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.705 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:32.705 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:32.705 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:32.705 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:32.706 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:32.706 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:32.706 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:32.706 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:32.706 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:32.707 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:32.707 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:32.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:32.708 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:32.708 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:32.708 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:32.708 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:32.708 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:32.709 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:32.709 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:32.709 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:32.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:32.709 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:32.710 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.710 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:32.710 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:32.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:32.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.741023_190.1", 2021-09-06T13:09:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:32.711 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.741023Z", 2021-09-06T13:09:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:32.713 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:32.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:32.714 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:32.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.741023_190.1 2021-09-06T13:09:37.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:37.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:37.718 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:37.768 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:37.768 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:37.886 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:37.887 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:37.887 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:37.887 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:37.887 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:37.888 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:37.888 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:37.888 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:37.888 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:37.888 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:37.889 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:37.889 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:37.889 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:37.889 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:37.890 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:37.890 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:37.890 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:37.890 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:37.890 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:37.891 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.891 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:37.891 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:37.891 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:37.892 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:37.892 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:37.892 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:37.892 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:37.892 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:37.893 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:37.893 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:37.893 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:37.893 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:37.894 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:37.894 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:37.894 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:37.894 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:37.895 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:37.895 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:37.895 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:37.895 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:37.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:37.896 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:37.896 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.896 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:37.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:37.896 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:09:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:37.897 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:37.898 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:37.898 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:09:37.898 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:37.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.899 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:37.899 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:37.899 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:37.899 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:37.899 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:37.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:37.900 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:37.901 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.599872_181.1 2021-09-06T13:09:42.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:42.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:42.904 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:42.949 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:43.190 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:43.190 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:43.190 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:43.190 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:43.191 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:43.191 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:43.191 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:43.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:43.192 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:43.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:43.192 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:43.192 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:43.192 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:43.193 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:43.194 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:43.194 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:43.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:43.194 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:43.195 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:43.196 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:43.196 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:43.196 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:43.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:43.197 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:43.197 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:43.197 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:43.197 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:43.197 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:43.198 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:43.198 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:43.198 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:43.198 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:43.198 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:43.199 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.199 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:43.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:43.199 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:43.199 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:43.200 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:43.200 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:09:43.200 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:43.200 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:43.200 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:43.201 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:09:43.201 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:43.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.201 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:43.202 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:43.202 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:43.202 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:43.202 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:43.202 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:43.203 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:43.204 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.599872_181.1 2021-09-06T13:09:48.205 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:48.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:48.206 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:48.269 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:48.479 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:48.480 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:48.480 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:48.480 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:48.481 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:48.481 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:48.481 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:48.481 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:48.482 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:48.483 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:48.483 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:48.483 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:48.483 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:48.483 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:48.484 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:48.484 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.484 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:48.484 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:48.484 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:48.485 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:48.485 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:48.485 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:48.485 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:48.485 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:48.486 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:48.486 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:48.486 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:48.486 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.486 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:48.487 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:48.488 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:48.488 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:48.488 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:48.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:48.488 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:48.489 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.489 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:48.489 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:48.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:48.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:48.490 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:48.490 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:09:48.490 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:48.490 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:48.491 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:48.491 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:09:48.491 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:48.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.492 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:48.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:48.492 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:48.492 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:48.492 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:48.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:48.493 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:48.493 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:09:48.494 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T13:09:48.374+0000 7f04fd82c700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T13:09:53.495 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:53.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:53.497 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:53.598 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:53.598 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:53.977 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:53.977 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:53.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:53.979 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:53.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:53.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:53.982 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:53.983 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:53.984 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:53.985 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:53.987 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:09:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:53.988 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:09:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:53.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:53.990 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:53.990 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:53.992 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:09:58.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:58.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:09:58.995 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:09:59.082 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:09:59.082 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:09:59.584 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:09:59.585 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:09:59.586 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:59.587 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:09:59.588 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:59.589 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:09:59.590 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:09:59.591 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:59.592 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:09:59.593 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:59.594 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:09:59.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:09:59.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:09:59.598 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:09:59.599 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:09:59.600 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:10:04.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:04.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:04.603 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:04.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:04.658 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:05.256 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:05.256 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:05.256 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:05.256 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:05.257 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:05.257 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:05.257 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:05.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:05.258 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:05.258 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:05.258 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:05.258 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:05.259 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:05.259 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:05.259 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:05.259 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:05.260 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:05.260 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:05.260 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:05.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.261 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:05.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:05.261 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:05.261 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:05.262 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:05.262 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:05.262 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:05.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:05.263 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:05.263 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:05.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:05.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:05.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:05.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:05.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:05.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:05.265 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:05.265 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:05.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:05.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:05.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:05.266 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:05.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.266 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:05.266 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:05.266 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:05.267 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:05.267 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:05.267 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:10:05.268 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:05.268 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:05.268 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:05.268 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:10:05.268 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:05.269 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.269 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.269 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:05.269 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:05.269 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:05.270 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:05.270 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:05.270 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:05.270 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:05.271 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:10:10.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:10.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:10.274 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:10.329 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:10.329 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:10.759 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:10.759 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:10.759 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:10.760 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:10.760 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:10.760 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:10.760 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:10.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:10.761 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:10.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:10.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:10.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:10.762 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:10.762 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:10.762 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:10.762 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:10.763 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:10.763 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:10.763 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:10.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:10.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:10.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:10.764 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:10.765 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:10.765 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:10.765 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:10.765 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:10.765 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:10.766 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:10.766 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:10.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:10.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:10.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:10.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:10.767 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:10.767 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:10.768 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:10.768 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:10.768 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:10.768 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:10.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:10.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:10.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:10.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:10.770 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:10.770 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:10.770 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:10:10.770 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:10.771 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:10.771 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:10.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:10:10.771 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:10.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.772 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:10.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:10.772 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:10.773 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:10.773 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:10.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:10.773 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:10.774 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.457681_169.1 2021-09-06T13:10:15.776 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:15.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:15.778 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:15.833 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:15.834 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:15.914 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:15.919 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:15.920 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:15.920 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:15.920 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:15.920 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:15.921 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:15.921 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:15.921 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:15.921 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:15.922 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:15.922 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:15.922 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:15.922 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:15.923 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:15.923 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:15.923 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:15.923 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:15.924 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:15.924 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.924 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:15.924 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:15.924 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:15.925 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:15.925 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:15.925 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:15.925 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:15.926 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:15.926 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:15.926 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:15.926 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:15.927 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.927 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:15.927 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:15.927 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:15.928 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:15.928 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:15.928 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:15.928 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:15.928 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:15.929 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:15.929 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:15.929 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:15.929 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.930 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:15.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:15.930 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:15.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:15.931 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:15.931 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:10:15.931 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:15.931 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:15.931 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:15.932 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:10:15.932 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:15.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.933 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:15.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:15.933 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:15.933 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:15.933 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:15.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:15.934 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:15.935 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:10:20.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:20.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:20.938 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:20.983 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:20.984 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:21.050 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:21.051 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:21.052 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:21.053 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:21.054 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.055 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:21.056 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:21.057 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:21.058 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:21.059 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:21.060 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:21.061 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:10:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:21.062 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:10:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:21.063 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:21.064 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:21.065 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:21.065 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:21.066 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.510450_174.1 2021-09-06T13:10:26.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:26.069 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:26.069 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:26.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:26.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:26.558 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:26.558 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:26.559 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:26.559 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:26.559 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:26.559 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:26.560 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:26.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:26.560 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:26.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:26.561 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:26.561 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:26.561 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:26.561 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:26.562 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:26.562 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:26.562 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:26.562 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:26.563 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:26.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:26.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:26.564 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:26.565 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:26.565 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:26.565 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:26.565 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:26.565 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:26.566 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.566 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:26.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:26.566 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:26.567 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:26.567 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:26.567 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:26.567 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:26.567 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:26.568 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:26.568 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:26.568 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:26.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:26.569 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:26.569 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:26.569 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:26.569 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:26.570 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:10:26.570 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:26.570 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:26.570 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:26.571 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:10:26.571 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:26.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.572 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:26.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:26.572 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:26.572 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:26.572 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:26.573 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:26.573 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:26.574 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.595266_180.1 2021-09-06T13:10:31.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:31.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:31.577 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:31.620 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:31.620 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:31.765 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:31.765 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:31.766 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:31.766 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:31.766 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:31.766 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:31.767 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:31.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:31.767 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:31.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:31.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:31.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:31.768 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:31.768 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:31.769 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:31.769 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:31.769 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:31.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:31.770 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:31.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.770 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:31.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:31.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:31.771 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:31.771 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:31.771 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:31.772 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:31.772 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:31.772 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:31.772 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:31.773 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:31.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.773 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:31.773 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:31.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:31.774 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:31.774 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:31.774 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:31.775 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:31.775 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:31.775 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:31.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:31.776 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:31.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:31.776 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:31.777 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:31.777 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:31.777 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:31.777 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:10:31.777 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:31.778 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:31.778 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:31.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:10:31.778 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:31.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.779 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:31.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:31.780 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:31.780 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:31.780 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:31.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:31.781 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:31.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.523913_177.1 2021-09-06T13:10:36.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:36.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:36.784 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:36.875 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:36.875 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:37.294 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:37.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:37.304 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:37.304 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:37.304 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:37.305 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:37.305 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:10:37.305 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:37.305 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:37.305 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:37.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:10:37.306 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:37.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.306 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:37.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:37.308 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:37.309 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:10:42.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:42.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:42.311 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:42.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:42.363 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:42.436 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:42.436 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:42.437 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:42.438 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:42.439 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:42.439 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:42.439 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:42.439 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:42.440 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:42.440 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:42.440 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:42.440 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:42.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:42.441 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:42.441 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:42.441 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:42.441 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:42.442 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:42.442 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:42.442 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:42.442 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:42.443 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:42.443 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:42.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.443 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:42.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:42.444 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:42.444 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:42.444 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:42.444 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:42.445 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:42.445 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:42.445 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:42.445 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:42.445 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:42.446 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:42.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:42.446 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:42.447 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:42.447 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:42.447 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:10:42.447 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:42.447 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:42.448 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:42.448 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:10:42.448 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:42.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.449 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:42.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:42.449 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:42.449 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:42.450 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:42.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:42.450 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:42.451 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:10:47.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:47.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:47.454 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:47.500 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:47.501 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:47.569 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:47.569 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:47.573 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:47.573 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:47.573 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:47.573 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:47.573 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:47.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.574 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:47.574 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:47.574 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:47.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:47.575 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:47.575 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:47.575 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:47.576 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:47.576 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:47.576 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:47.576 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:47.576 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:47.577 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:47.577 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:47.577 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:47.578 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:47.578 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:47.578 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:47.578 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:47.579 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:47.579 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:47.579 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:47.579 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:47.580 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:47.580 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:47.580 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:47.580 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:10:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:47.581 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:10:47.582 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:47.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.582 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.582 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:47.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:47.583 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:47.583 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:47.583 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:47.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:47.584 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:47.585 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:10:52.586 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:52.587 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:52.588 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:52.668 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:52.669 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:52.735 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:52.736 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:52.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:52.738 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:52.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:52.740 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:52.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:52.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:52.743 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:52.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:52.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:10:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:52.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:10:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.747 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:52.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:52.749 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:52.750 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:10:57.751 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:57.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:10:57.753 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:10:57.992 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:10:57.992 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:10:58.070 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:10:58.071 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:10:58.071 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:10:58.072 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:10:58.072 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:10:58.072 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:10:58.072 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:58.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:58.073 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:10:58.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:58.073 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:10:58.073 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:58.074 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:58.074 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:10:58.074 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:58.074 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:10:58.075 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:58.075 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:10:58.075 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:58.075 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:58.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:58.076 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:10:58.076 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:58.077 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:58.077 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:10:58.077 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:58.077 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:58.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:58.078 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:10:58.078 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:58.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.078 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:58.079 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:58.079 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:10:58.079 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:58.079 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:58.079 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:10:58.080 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:58.080 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:58.080 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:58.080 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:10:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:10:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:10:58.082 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:10:58.082 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:10:58.082 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:10:58.082 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:10:58.082 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:10:58.083 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:10:58.083 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:10:58.083 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:10:58.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.084 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:10:58.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:10:58.084 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:10:58.084 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:10:58.084 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:10:58.085 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:10:58.085 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:10:58.086 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:11:03.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:03.089 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:03.089 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:03.159 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:03.159 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:03.520 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:03.520 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:03.520 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:03.521 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:03.521 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:03.521 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:03.521 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:03.521 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:03.522 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:03.522 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:03.522 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:03.522 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:03.523 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:03.523 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:03.523 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:03.523 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:03.524 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:03.524 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:03.524 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:03.524 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:03.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:03.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:03.525 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:03.525 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:03.525 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:03.526 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:03.526 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:03.526 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:03.526 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:03.526 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:03.527 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:03.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:03.527 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:03.527 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:03.528 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:03.528 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:03.528 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:03.528 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:03.528 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:03.529 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:03.529 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:03.529 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.529 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:03.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:03.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:03.530 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:03.530 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:03.530 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:11:03.530 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:03.531 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:03.531 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:03.531 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:11:03.531 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:03.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.532 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:03.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:03.532 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:03.532 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:03.533 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:03.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:03.533 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:03.534 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.471383_170.1 2021-09-06T13:11:08.536 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:08.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:08.537 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:08.628 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:08.628 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:08.740 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:08.741 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:08.741 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:08.741 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:08.741 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:08.742 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:08.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:08.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:08.742 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:08.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:08.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:08.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:08.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:08.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:08.744 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:08.744 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:08.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:08.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:08.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:08.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:08.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:08.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:08.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:08.746 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:08.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:08.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:08.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:08.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:08.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:08.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:08.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:08.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:08.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:08.748 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:08.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:08.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:08.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:08.749 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:08.749 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:08.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:08.750 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:08.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.750 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:08.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:08.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:08.751 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:08.751 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:08.751 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:11:08.751 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:08.752 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:08.752 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:08.752 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:11:08.752 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:08.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.753 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:08.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:08.753 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:08.753 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:08.754 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:08.754 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:08.754 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:08.755 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:11:13.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:13.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:13.758 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:13.812 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:13.812 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:13.938 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:13.938 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:13.939 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:13.940 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:13.941 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:13.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:13.943 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:13.945 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:13.946 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:13.947 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:13.948 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:11:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:13.949 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:11:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:13.950 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:13.951 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:13.952 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:13.952 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:13.953 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:11:18.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:18.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:18.956 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:19.006 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:19.006 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:19.079 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:19.079 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:19.080 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:19.080 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:19.080 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:19.080 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:19.081 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:19.082 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:19.083 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:19.084 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:19.085 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:19.086 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.087 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:19.088 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:19.089 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:19.090 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:19.091 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:11:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:19.092 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:11:19.093 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:19.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.093 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:19.094 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:19.095 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:19.096 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:11:24.097 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:24.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:24.098 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:24.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:24.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:24.309 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:24.310 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:24.310 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:24.310 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:24.310 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:24.311 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:24.311 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:24.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:24.311 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:24.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:24.312 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:24.312 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:24.312 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:24.312 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:24.313 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:24.313 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:24.313 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:24.313 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:24.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:24.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:24.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:24.314 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:24.315 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:24.315 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:24.315 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:24.315 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:24.315 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:24.316 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:24.316 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:24.316 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:24.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.316 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:24.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:24.317 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:24.317 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:24.317 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:24.317 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:24.318 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:24.318 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:24.318 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:24.318 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:24.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:24.320 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:24.320 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:24.320 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:11:24.320 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:24.320 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:24.321 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:24.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:11:24.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:24.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.322 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:24.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:24.322 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:24.322 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:24.322 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:24.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:24.323 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:24.324 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.523913_177.1 2021-09-06T13:11:29.325 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:29.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:29.327 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:29.370 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:29.370 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:29.437 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:29.437 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:29.437 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:29.438 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:29.438 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:29.438 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:29.438 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:29.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:29.439 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:29.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:29.439 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:29.440 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:29.440 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:29.440 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:29.440 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:29.440 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:29.441 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:29.441 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:29.441 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:29.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:29.442 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:29.442 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:29.442 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:29.442 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:29.443 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:29.443 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:29.443 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:29.443 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:29.443 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:29.444 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:29.444 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.444 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:29.444 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:29.444 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:29.445 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:29.445 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:29.445 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:29.445 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:29.446 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:29.446 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:29.446 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:29.446 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:29.447 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.447 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:29.447 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:29.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:29.448 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:29.448 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:29.448 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:11:29.448 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:29.449 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:29.449 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:29.449 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:11:29.449 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:29.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.450 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.450 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:29.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:29.450 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:29.451 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:29.451 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:29.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:29.451 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:29.452 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:11:34.454 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:34.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:34.455 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:34.512 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:34.973 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:34.973 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:34.974 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:34.974 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:34.974 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:34.974 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:34.975 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:34.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:34.975 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:34.975 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:34.976 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:34.976 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:34.976 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:34.976 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:34.977 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:34.977 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:34.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:34.977 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:34.977 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:34.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:34.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:34.978 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:34.978 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:34.979 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:34.979 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:34.979 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:34.979 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:34.980 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:34.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:34.980 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:34.980 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:34.981 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:34.982 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:34.982 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:34.982 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:34.982 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:34.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:34.983 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:34.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:34.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:34.983 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:34.984 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:34.984 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:34.984 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:11:34.984 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:34.984 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:34.985 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:34.985 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:11:34.985 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:34.985 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.986 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:34.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:34.986 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:34.986 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:34.987 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:34.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:34.987 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:34.988 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:11:39.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:39.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:39.991 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:40.043 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:40.044 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:40.110 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:40.110 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:40.110 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:40.111 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:40.111 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:40.111 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:40.111 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:40.112 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:40.112 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:40.112 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:40.112 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:40.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:40.113 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:40.113 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:40.113 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:40.113 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:40.114 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:40.114 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:40.114 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:40.114 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.115 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:40.115 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:40.115 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:40.115 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:40.115 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:40.116 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:40.116 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:40.116 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:40.116 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:40.117 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:40.117 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:40.117 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.117 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:40.117 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:40.118 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:40.118 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:40.118 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:40.118 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:40.119 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:40.119 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:40.119 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:40.119 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:40.119 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:40.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.120 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:40.120 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:40.120 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:40.121 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:40.121 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:40.121 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.016042_207.1", 2021-09-06T13:11:40.121 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:40.121 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:40.122 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:40.122 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.016042Z", 2021-09-06T13:11:40.122 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:40.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.122 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.123 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:40.123 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:40.123 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:40.123 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:40.124 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:40.124 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:40.124 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:40.125 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932662.016042_207.1 2021-09-06T13:11:45.127 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:45.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:45.128 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:45.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:45.181 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:45.248 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:45.248 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:45.248 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:45.249 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:45.249 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:45.249 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:45.249 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:45.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:45.250 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:45.250 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:45.250 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:45.250 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:45.251 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:45.251 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:45.251 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:45.251 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:45.252 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:45.252 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:45.252 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:45.252 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:45.253 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:45.253 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:45.253 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:45.253 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:45.254 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:45.254 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:45.254 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:45.254 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:45.255 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:45.255 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:45.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:45.256 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:45.256 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:45.257 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:45.257 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:45.257 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:45.257 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:45.258 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:45.258 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:45.258 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:45.258 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:45.258 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:45.259 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:45.259 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:45.259 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:45.259 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:45.260 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:11:45.260 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:45.260 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:45.260 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:45.261 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:11:45.261 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:45.261 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.261 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.261 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:45.262 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:45.262 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:45.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:45.262 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:45.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:45.263 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:45.264 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.510450_174.1 2021-09-06T13:11:50.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:50.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:50.267 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:50.313 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:50.313 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:50.637 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:50.638 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:50.638 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:50.638 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:50.639 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:50.639 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:50.639 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:50.639 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:50.640 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:50.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:50.640 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:50.640 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:50.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:50.641 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:50.641 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:50.641 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:50.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:50.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:50.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:50.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:50.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:50.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:50.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:50.643 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:50.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:50.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:50.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:50.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:50.645 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:50.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:50.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:50.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:50.646 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:50.646 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:50.646 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:50.646 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:50.646 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:50.647 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:50.647 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:50.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:50.647 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:50.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:50.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:50.648 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:50.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:50.649 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:50.649 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:11:50.649 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:50.649 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:50.650 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:50.650 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:11:50.650 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:50.650 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.651 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:50.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:50.651 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:50.651 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:50.652 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:50.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:50.652 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:50.652 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.517790_176.1 2021-09-06T13:11:55.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:55.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:11:55.656 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:11:55.766 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:11:55.766 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:11:56.078 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:11:56.079 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:11:56.079 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:11:56.079 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:11:56.079 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:11:56.080 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:11:56.080 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:56.080 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:56.080 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:11:56.080 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:56.081 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:11:56.081 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:56.081 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:56.081 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:11:56.082 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:56.082 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:11:56.082 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:56.082 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:11:56.083 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:56.083 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.083 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:56.083 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:56.084 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:11:56.084 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:56.084 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:56.084 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:11:56.085 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:56.085 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:56.085 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:56.085 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:11:56.086 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:56.086 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.086 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:56.086 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:56.086 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:11:56.087 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:56.087 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:56.087 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:11:56.087 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:11:56.088 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:56.089 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.089 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:56.089 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:11:56.089 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:11:56.089 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:11:56.090 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:11:56.090 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.453043_167.1", 2021-09-06T13:11:56.090 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:11:56.090 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:11:56.091 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:11:56.091 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.453043Z", 2021-09-06T13:11:56.091 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:11:56.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.091 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.092 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:11:56.092 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:11:56.092 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:11:56.092 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:11:56.092 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:11:56.093 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:11:56.093 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:11:56.094 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.453043_167.1 2021-09-06T13:12:01.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:01.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:01.096 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:01.193 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:01.194 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:01.746 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:01.746 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:01.747 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:01.747 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:01.747 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:01.747 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:01.748 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:01.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:01.748 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:01.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:01.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:01.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:01.749 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:01.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:01.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:01.750 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:01.750 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:01.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:01.750 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:01.751 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:01.751 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:01.751 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:01.752 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:01.752 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:01.752 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:01.752 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:01.753 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:01.753 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:01.753 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:01.753 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:01.753 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.754 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:01.754 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:01.754 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:01.754 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:01.755 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:01.755 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:01.755 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:01.755 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:01.756 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:01.756 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:01.756 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:01.756 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:01.757 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:01.757 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:01.757 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:01.757 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:01.758 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:12:01.758 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:01.758 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:01.758 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:01.759 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:12:01.759 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:01.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.759 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.760 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:01.760 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:01.760 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:01.760 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:01.760 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:01.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:01.761 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:01.762 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.476870_172.1 2021-09-06T13:12:06.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:06.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:06.765 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:06.851 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:06.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:07.277 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:07.278 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:07.278 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:07.278 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:07.279 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:07.279 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:07.279 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:07.279 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:07.279 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:07.280 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:07.280 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:07.280 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:07.280 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:07.281 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:07.281 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:07.281 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:07.281 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:07.281 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:07.282 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:07.282 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.282 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:07.282 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:07.283 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:07.283 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:07.283 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:07.283 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:07.283 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:07.284 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:07.284 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:07.284 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:07.284 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:07.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.285 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:07.285 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:07.285 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:07.285 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:07.286 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:07.286 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:07.286 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:07.286 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:07.287 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:07.287 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:07.287 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:07.287 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.287 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:07.288 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:07.288 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:07.288 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:07.288 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:07.288 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:12:07.289 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:07.289 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:07.289 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:07.289 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:12:07.289 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:07.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.290 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.290 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:07.290 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:07.291 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:07.291 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:07.291 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:07.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:07.291 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:07.293 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.483094_173.1 2021-09-06T13:12:09.240 INFO:tasks.rgw.c2.client.0.smithi052.stdout:2021-09-06T13:12:09.239+0000 7f04fc029700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T13:12:09.246 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T13:12:09.245+0000 7f89dff66700 -1 ERROR: failed operation: ERROR: collect() returned error (ret=-5) 2021-09-06T13:12:12.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:12.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:12.295 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:12.360 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:12.360 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:12.888 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:12.889 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:12.890 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:12.891 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:12.892 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:12.893 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:12.894 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.895 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:12.896 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.737368_189.1", 2021-09-06T13:12:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.737368Z", 2021-09-06T13:12:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:12.903 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:12.904 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.737368_189.1 2021-09-06T13:12:17.905 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:17.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:17.907 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:17.954 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:17.955 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:18.027 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:18.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:18.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:18.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:18.031 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:18.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:18.033 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:18.034 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:18.035 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:18.036 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:18.038 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:12:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:12:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:18.042 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:18.043 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.523913_177.1 2021-09-06T13:12:23.044 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:23.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:23.046 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:23.096 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:23.096 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:23.171 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:23.172 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:23.172 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:23.172 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:23.172 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:23.173 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:23.173 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:23.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:23.173 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:23.174 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:23.174 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:23.174 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:23.174 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:23.175 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:23.176 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:23.176 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:23.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:23.177 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:23.177 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:23.177 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:23.177 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:23.177 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:23.178 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:23.178 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:23.178 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:23.178 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:23.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.179 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:23.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:23.179 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:23.179 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:23.180 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:23.180 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:23.180 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:23.180 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:23.180 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:23.181 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:23.181 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:23.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:23.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:23.182 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:23.182 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:23.182 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:23.183 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.454323_168.1", 2021-09-06T13:12:23.183 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:23.183 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:23.183 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.454323Z", 2021-09-06T13:12:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:23.186 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:23.187 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.454323_168.1 2021-09-06T13:12:28.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:28.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:28.190 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:28.279 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:28.280 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:28.567 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:28.568 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:28.568 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:28.568 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:28.569 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:28.569 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:28.569 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:28.569 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:28.569 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:28.570 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:28.570 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:28.570 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:28.570 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:28.571 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:28.571 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:28.571 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:28.571 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:28.571 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:28.572 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:28.572 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:28.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:28.573 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:28.573 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:28.573 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:28.573 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:28.573 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:28.574 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:28.574 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:28.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:28.574 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:28.575 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:28.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:28.575 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:28.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:28.576 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:28.576 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:28.576 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:28.576 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:28.577 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:28.577 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:28.577 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:28.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.577 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:28.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:28.578 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:28.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:28.578 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:28.578 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:12:28.579 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:28.579 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:28.579 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:28.579 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:12:28.580 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:28.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.580 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:28.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:28.581 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:28.581 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:28.581 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:28.581 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:28.581 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:28.583 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.599872_181.1 2021-09-06T13:12:33.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:33.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:12:33.587 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:12:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:33.653 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:33.924 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:33.924 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:12:33.924 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:12:33.925 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:12:33.925 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:12:33.925 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:33.925 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:33.925 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:33.926 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:12:33.926 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:33.926 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:12:33.926 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:33.927 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:33.927 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:12:33.927 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:33.927 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:12:33.928 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:33.928 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:12:33.928 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:33.928 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.928 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:33.929 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:33.929 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:12:33.929 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:33.929 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:33.930 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:12:33.930 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:33.930 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:33.930 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:33.930 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:12:33.931 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:33.931 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.931 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:33.931 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:33.932 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:12:33.932 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:33.932 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:33.932 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:12:33.932 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:33.933 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:33.933 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:33.933 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:12:33.933 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:12:33.934 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:33.935 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:12:33.935 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:12:33.935 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:12:33.935 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:12:33.936 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:12:33.936 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:12:33.936 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:33.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.936 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.937 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:33.937 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:33.937 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:12:33.937 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:12:33.937 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:12:33.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:33.938 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:33.938 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630933633.195130_278.1 target=1_1630932661.650194_182.1 2021-09-06T13:12:38.941 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... FAIL 2021-09-06T13:12:38.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:12:38.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:12:38.944 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T13:12:39.032 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:39.032 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:39.204 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T13:12:39.205+0000 7fc73ba7e380 0 failed reading obj info from .rgw.root:zone_info.4561051f-a4da-4b2a-84e5-f27920586526: (2) No such file or directory 2021-09-06T13:12:39.205 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T13:12:39.205+0000 7fc73ba7e380 0 WARNING: could not read zone params for zone id=4561051f-a4da-4b2a-84e5-f27920586526 name=test-zone3 2021-09-06T13:12:39.206 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T13:12:39.207+0000 7fc73ba7e380 0 failed reading obj info from .rgw.root:zone_info.a488f9d2-4d4c-49b4-b839-c5d73c4d316e: (2) No such file or directory 2021-09-06T13:12:39.207 INFO:teuthology.orchestra.run.smithi052.stderr:2021-09-06T13:12:39.207+0000 7fc73ba7e380 0 WARNING: could not read zone params for zone id=a488f9d2-4d4c-49b4-b839-c5d73c4d316e name=test-zone1 2021-09-06T13:12:39.230 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:39.230 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:12:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T13:12:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "domain_root": "test-zone2.rgw.meta:root", 2021-09-06T13:12:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "control_pool": "test-zone2.rgw.control", 2021-09-06T13:12:39.231 INFO:teuthology.orchestra.run.smithi052.stdout: "gc_pool": "test-zone2.rgw.log:gc", 2021-09-06T13:12:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "lc_pool": "test-zone2.rgw.log:lc", 2021-09-06T13:12:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "log_pool": "test-zone2.rgw.log", 2021-09-06T13:12:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "intent_log_pool": "test-zone2.rgw.log:intent", 2021-09-06T13:12:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "usage_log_pool": "test-zone2.rgw.log:usage", 2021-09-06T13:12:39.232 INFO:teuthology.orchestra.run.smithi052.stdout: "roles_pool": "test-zone2.rgw.meta:roles", 2021-09-06T13:12:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "reshard_pool": "test-zone2.rgw.log:reshard", 2021-09-06T13:12:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "user_keys_pool": "test-zone2.rgw.meta:users.keys", 2021-09-06T13:12:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "user_email_pool": "test-zone2.rgw.meta:users.email", 2021-09-06T13:12:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "user_swift_pool": "test-zone2.rgw.meta:users.swift", 2021-09-06T13:12:39.233 INFO:teuthology.orchestra.run.smithi052.stdout: "user_uid_pool": "test-zone2.rgw.meta:users.uid", 2021-09-06T13:12:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "otp_pool": "test-zone2.rgw.otp", 2021-09-06T13:12:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "system_key": { 2021-09-06T13:12:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "access_key": "IH9RGB4MJICQ8BMT", 2021-09-06T13:12:39.234 INFO:teuthology.orchestra.run.smithi052.stdout: "secret_key": "ukc6mrSyiz7SPjYiXYAnaYM9tRpZwEeu" 2021-09-06T13:12:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_pools": [ 2021-09-06T13:12:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "default-placement", 2021-09-06T13:12:39.235 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:12:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "index_pool": "test-zone2.rgw.buckets.index", 2021-09-06T13:12:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": { 2021-09-06T13:12:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD": { 2021-09-06T13:12:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "data_pool": "test-zone2.rgw.buckets.data", 2021-09-06T13:12:39.236 INFO:teuthology.orchestra.run.smithi052.stdout: "compression_type": "random" 2021-09-06T13:12:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "data_extra_pool": "test-zone2.rgw.buckets.non-ec", 2021-09-06T13:12:39.237 INFO:teuthology.orchestra.run.smithi052.stdout: "index_type": 0 2021-09-06T13:12:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:12:39.238 INFO:teuthology.orchestra.run.smithi052.stdout: "notif_pool": "test-zone2.rgw.log:notif" 2021-09-06T13:12:39.239 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:39.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:12:39.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:12:39.241 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-09-06T13:12:39.313 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:12:39.314 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:12:39.466 INFO:teuthology.orchestra.run.smithi052.stderr:Sending period to new master zone a488f9d2-4d4c-49b4-b839-c5d73c4d316e 2021-09-06T13:12:40.508 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:12:40.508 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:40.509 INFO:teuthology.orchestra.run.smithi052.stdout: "epoch": 5, 2021-09-06T13:12:40.509 INFO:teuthology.orchestra.run.smithi052.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T13:12:40.509 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": [], 2021-09-06T13:12:40.509 INFO:teuthology.orchestra.run.smithi052.stdout: "period_map": { 2021-09-06T13:12:40.510 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:12:40.510 INFO:teuthology.orchestra.run.smithi052.stdout: "zonegroups": [ 2021-09-06T13:12:40.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.510 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "test-zonegroup", 2021-09-06T13:12:40.511 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zonegroup", 2021-09-06T13:12:40.511 INFO:teuthology.orchestra.run.smithi052.stdout: "api_name": "", 2021-09-06T13:12:40.511 INFO:teuthology.orchestra.run.smithi052.stdout: "is_master": "true", 2021-09-06T13:12:40.511 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T13:12:40.512 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:12:40.512 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.512 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames": [], 2021-09-06T13:12:40.512 INFO:teuthology.orchestra.run.smithi052.stdout: "hostnames_s3website": [], 2021-09-06T13:12:40.512 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:12:40.513 INFO:teuthology.orchestra.run.smithi052.stdout: "zones": [ 2021-09-06T13:12:40.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.513 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:12:40.513 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone3", 2021-09-06T13:12:40.514 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T13:12:40.514 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8001" 2021-09-06T13:12:40.514 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.514 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T13:12:40.515 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T13:12:40.515 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:12:40.516 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T13:12:40.516 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T13:12:40.516 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T13:12:40.517 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T13:12:40.517 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T13:12:40.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.518 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.518 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:12:40.519 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone4", 2021-09-06T13:12:40.519 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T13:12:40.520 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2021-09-06T13:12:40.520 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.521 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T13:12:40.521 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T13:12:40.521 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:12:40.521 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T13:12:40.522 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "pubsub", 2021-09-06T13:12:40.522 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2021-09-06T13:12:40.523 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [ 2021-09-06T13:12:40.523 INFO:teuthology.orchestra.run.smithi052.stdout: "test-zone1" 2021-09-06T13:12:40.523 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T13:12:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.524 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:12:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone1", 2021-09-06T13:12:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T13:12:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:12:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.525 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T13:12:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T13:12:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:12:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T13:12:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T13:12:40.526 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "true", 2021-09-06T13:12:40.527 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T13:12:40.527 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T13:12:40.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.527 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.527 INFO:teuthology.orchestra.run.smithi052.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:12:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "test-zone2", 2021-09-06T13:12:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: "endpoints": [ 2021-09-06T13:12:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T13:12:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.528 INFO:teuthology.orchestra.run.smithi052.stdout: "log_meta": "false", 2021-09-06T13:12:40.529 INFO:teuthology.orchestra.run.smithi052.stdout: "log_data": "true", 2021-09-06T13:12:40.529 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:12:40.529 INFO:teuthology.orchestra.run.smithi052.stdout: "read_only": "false", 2021-09-06T13:12:40.529 INFO:teuthology.orchestra.run.smithi052.stdout: "tier_type": "", 2021-09-06T13:12:40.529 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from_all": "false", 2021-09-06T13:12:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_from": [], 2021-09-06T13:12:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: "redirect_zone": "" 2021-09-06T13:12:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.530 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: "placement_targets": [ 2021-09-06T13:12:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: "name": "default-placement", 2021-09-06T13:12:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: "tags": [], 2021-09-06T13:12:40.531 INFO:teuthology.orchestra.run.smithi052.stdout: "storage_classes": [ 2021-09-06T13:12:40.532 INFO:teuthology.orchestra.run.smithi052.stdout: "STANDARD" 2021-09-06T13:12:40.532 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:40.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.532 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.532 INFO:teuthology.orchestra.run.smithi052.stdout: "default_placement": "default-placement", 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_policy": { 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: "groups": [] 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.533 INFO:teuthology.orchestra.run.smithi052.stdout: ], 2021-09-06T13:12:40.534 INFO:teuthology.orchestra.run.smithi052.stdout: "short_zone_ids": [ 2021-09-06T13:12:40.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.535 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:12:40.535 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3257390013 2021-09-06T13:12:40.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:12:40.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 2300744410 2021-09-06T13:12:40.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:12:40.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 554922684 2021-09-06T13:12:40.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:12:40.538 INFO:teuthology.orchestra.run.smithi052.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:12:40.538 INFO:teuthology.orchestra.run.smithi052.stdout: "val": 3217630970 2021-09-06T13:12:40.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.538 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:12:40.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T13:12:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:12:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "period_config": { 2021-09-06T13:12:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "bucket_quota": { 2021-09-06T13:12:40.539 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T13:12:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T13:12:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T13:12:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T13:12:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T13:12:40.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: "user_quota": { 2021-09-06T13:12:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: "enabled": false, 2021-09-06T13:12:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: "check_on_raw": false, 2021-09-06T13:12:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size": -1, 2021-09-06T13:12:40.541 INFO:teuthology.orchestra.run.smithi052.stdout: "max_size_kb": 0, 2021-09-06T13:12:40.542 INFO:teuthology.orchestra.run.smithi052.stdout: "max_objects": -1 2021-09-06T13:12:40.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:12:40.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:12:40.542 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:12:40.542 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_name": "test-realm", 2021-09-06T13:12:40.543 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:12:40.543 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:12:40.543 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-09-06T13:13:40.545 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-45 2021-09-06T13:13:40.623 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:40.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:40.624 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:13:40.678 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:13:40.678 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:13:40.744 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T13:13:40.763 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T13:13:40.764 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:13:40.764 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:13:40.764 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T13:13:40.764 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:13:40.764 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:13:40.765 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:13:40.765 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T13:13:40.765 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:13:40.765 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:13:40.766 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:13:40.766 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T13:13:40.766 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:13:40.766 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:13:40.766 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:13:40.767 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T13:13:40.575123Z" 2021-09-06T13:13:40.767 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:13:40.767 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T13:13:40.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T13:13:40.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:40.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:40.770 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:13:41.101 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:13:41.101 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:13:41.668 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:13:41.668 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:13:41.669 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:13:41.669 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:13:41.669 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:13:41.669 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:13:41.670 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:41.670 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:41.670 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:13:41.670 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:41.671 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:13:41.671 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:41.671 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:41.671 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:13:41.671 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:41.672 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:13:41.672 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:41.672 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:13:41.672 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:41.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:41.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:41.673 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:13:41.673 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:41.674 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:41.674 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:13:41.674 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:41.674 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:41.675 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:41.675 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:13:41.675 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:41.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:41.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:41.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:13:41.676 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:41.676 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:41.677 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:13:41.677 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:41.677 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:41.677 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:41.677 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:13:41.678 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:41.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:41.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:41.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:13:41.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:41.679 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:41.679 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:13:41.680 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:41.680 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:41.680 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:41.680 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:13:41.680 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:41.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.681 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:13:41.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:41.681 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:13:41.682 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:13:41.682 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:13:41.682 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:41.682 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:13:41.683 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:13:46.684 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:46.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:46.685 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:13:46.737 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:13:46.738 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:13:46.835 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:13:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:13:46.835 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:13:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:13:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:13:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:13:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:46.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:13:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:13:46.837 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:13:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:46.838 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:13:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:13:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:13:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:13:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:13:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:46.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:46.843 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:13:46.843 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:46.843 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:46.843 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:13:46.844 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:46.844 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:46.844 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:46.844 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:13:46.845 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:46.845 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.845 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:46.845 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:46.845 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:13:46.846 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:46.846 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:46.846 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:13:46.846 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:46.846 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:46.847 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:46.847 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:13:46.847 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:46.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.848 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:13:46.848 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:46.848 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:13:46.848 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:13:46.849 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:13:46.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:46.849 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:13:46.850 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:13:51.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:51.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:51.853 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:13:51.902 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:13:51.902 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:13:51.975 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:13:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:13:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:13:51.976 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:13:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:13:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:13:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:51.977 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:13:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:13:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:51.978 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:13:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:13:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:51.979 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:13:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:51.980 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:13:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:51.981 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:13:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:51.982 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:13:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:51.983 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:13:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:13:51.984 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:13:51.985 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:51.986 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:13:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:13:51.987 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:13:51.988 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:13:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:51.989 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:13:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:13:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:13:51.990 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:51.990 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:13:51.992 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:13:56.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:56.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:13:56.994 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:13:57.069 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:13:57.070 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:13:57.194 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:13:57.194 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:13:57.194 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:13:57.195 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:13:57.195 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:13:57.195 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:13:57.195 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:57.196 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:57.196 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:13:57.196 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:57.196 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:13:57.196 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:57.197 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:57.197 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:13:57.197 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:57.197 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:13:57.197 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:57.198 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:13:57.198 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:57.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:57.199 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:57.199 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:13:57.199 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:57.199 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:57.199 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:13:57.200 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:57.200 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:57.200 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:57.200 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:13:57.201 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:57.201 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.201 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:57.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:57.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:13:57.202 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:57.202 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:57.202 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:13:57.202 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:57.202 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:57.203 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:57.203 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:13:57.203 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:57.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:57.204 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:13:57.204 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:13:57.204 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:13:57.204 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:13:57.205 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:13:57.205 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:13:57.205 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:13:57.205 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:13:57.205 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:13:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:13:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:13:57.206 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:13:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:13:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:13:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:13:57.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:13:57.207 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:13:57.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:02.210 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:02.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:02.211 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:02.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:02.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:02.799 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:02.800 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:02.801 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:02.802 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:02.803 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:02.803 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:02.803 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:02.803 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:02.804 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:02.804 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:02.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:02.804 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:02.805 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:02.805 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:02.805 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:02.805 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:02.805 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:02.806 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:02.806 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:02.806 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:02.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:02.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:02.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:02.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:02.807 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:02.807 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:02.808 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:02.808 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:02.808 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:02.808 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:02.808 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:02.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:02.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:02.809 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:02.809 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:02.810 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:02.810 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:02.810 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:02.810 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:02.811 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:02.811 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:02.811 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:02.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.812 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:02.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:02.812 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:02.812 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:02.812 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:02.813 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:02.813 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:02.813 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:07.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:07.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:07.816 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:07.866 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:07.946 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:07.947 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:07.947 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:07.947 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:07.949 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:07.949 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:07.949 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:07.949 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:07.950 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:07.950 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:07.950 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:07.950 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:07.951 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:07.951 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:07.951 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.951 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:07.951 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:07.952 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:07.952 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:07.952 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:07.952 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:07.952 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:07.953 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:07.953 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:07.953 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:07.953 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:07.953 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:07.954 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:07.955 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:07.955 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:07.955 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:07.955 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:07.956 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:07.956 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:07.956 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.956 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:07.956 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:07.957 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:07.957 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:07.957 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:07.957 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:07.957 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:07.958 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:07.958 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:07.958 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:07.958 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:07.958 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.959 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.959 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:07.959 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:07.959 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:07.959 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:07.960 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:07.960 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:07.960 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:07.961 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:11.719 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T13:14:11.718+0000 7f89f6793700 -1 rgw async rados processor: ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-09-06T13:14:12.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:12.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:12.964 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:13.065 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:13.065 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:13.347 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:13.347 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:13.348 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:13.348 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:13.348 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:13.348 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:13.349 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:13.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:13.349 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:13.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:13.349 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:13.350 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:13.350 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:13.350 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:13.350 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:13.351 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:13.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:13.351 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:13.351 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:13.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.352 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:13.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:13.352 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:13.352 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:13.352 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:13.353 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:13.353 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:13.353 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:13.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:13.353 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:13.354 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:13.354 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:13.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:13.355 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:13.355 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:13.355 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:13.355 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:13.355 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:13.356 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:13.356 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:13.356 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:13.356 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:13.356 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:13.357 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:13.357 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:13.357 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:13.358 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:13.358 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:13.358 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:13.358 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:13.358 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:13.359 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:13.359 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:13.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.359 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.359 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:13.360 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:13.360 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:13.360 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:13.360 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:13.361 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:13.361 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:13.362 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:18.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:18.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:18.365 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:18.416 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:18.417 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:18.490 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:18.490 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:18.491 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:18.491 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:18.491 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:18.491 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:18.491 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:18.492 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:18.492 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:18.492 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:18.492 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:18.493 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:18.493 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:18.493 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:18.493 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:18.493 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:18.494 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:18.494 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:18.494 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:18.494 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:18.495 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:18.495 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:18.495 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:18.495 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:18.495 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:18.496 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:18.496 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:18.496 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:18.496 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:18.496 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:18.497 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:18.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:18.497 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:18.498 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:18.501 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:18.501 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:18.501 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:18.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:18.501 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:18.502 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:18.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.502 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:18.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:18.503 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:18.503 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:18.503 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:18.503 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:18.504 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:18.505 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:23.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:23.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:23.508 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:23.558 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:23.631 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:23.632 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:23.632 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:23.632 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:23.632 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:23.633 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:23.633 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:23.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:23.633 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:23.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:23.634 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:23.634 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:23.634 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:23.634 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:23.634 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:23.635 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:23.635 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:23.635 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:23.635 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:23.636 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:23.637 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:23.638 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:23.639 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:23.640 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:23.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:23.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.643 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:23.644 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:23.645 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:23.646 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:28.647 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:28.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:28.648 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:28.744 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:28.744 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:29.066 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:29.068 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:29.068 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:29.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:29.068 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:29.068 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:29.069 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:29.069 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:29.069 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:29.069 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:29.069 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:29.070 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:29.070 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:29.070 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:29.070 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:29.070 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.071 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:29.071 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:29.071 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:29.071 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:29.072 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:29.072 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:29.072 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:29.072 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:29.072 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:29.073 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:29.073 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:29.073 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.073 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:29.073 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:29.074 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:29.074 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:29.074 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:29.074 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:29.075 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:29.075 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:29.075 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:29.075 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:29.075 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:29.076 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.076 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:29.076 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:29.076 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:29.076 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:29.077 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:29.077 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:29.077 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:29.077 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:29.077 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:29.078 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:29.078 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:29.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.078 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.079 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:29.079 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:29.079 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:29.079 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:29.079 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:29.080 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:29.080 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:29.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:34.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:34.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:34.083 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:34.154 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:34.154 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:34.247 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:34.247 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:34.248 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:34.249 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:34.250 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:34.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:34.252 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:34.253 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:34.254 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:34.255 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.256 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:34.257 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:34.258 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:34.259 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:34.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:34.260 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:34.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:39.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:39.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:39.264 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:39.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:39.362 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:39.729 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:39.729 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:39.730 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:39.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:39.732 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:39.733 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:39.734 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:39.735 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:39.736 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:39.737 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:39.738 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:39.738 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:39.738 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:39.738 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:39.739 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:39.739 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:39.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:39.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:39.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:39.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:39.740 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630933633.195130_278.1", 2021-09-06T13:14:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:39.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:07:13.195130Z", 2021-09-06T13:14:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.742 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:39.743 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:39.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:39.744 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:39.745 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630933633.195130_278.1 2021-09-06T13:14:44.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:44.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:44.748 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:44.810 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:44.810 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:45.337 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:45.337 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:45.337 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:45.337 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:45.338 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:45.338 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:45.338 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.338 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.338 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.339 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:45.340 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.340 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:45.340 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.340 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:45.340 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.341 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.341 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.341 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.341 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:45.341 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.342 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.342 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:45.342 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.342 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.342 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.343 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:45.343 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.343 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.343 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.344 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:45.344 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.344 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.344 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:45.344 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.345 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.346 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.346 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:45.346 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.346 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.347 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:14:45.347 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.347 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.347 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.347 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:13:40.575123Z", 2021-09-06T13:14:45.348 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.348 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:45.348 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.349 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:45.349 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:45.349 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:45.349 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.349 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:45.350 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T13:14:45.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T13:14:45.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:45.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:45.352 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:45.403 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:45.403 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:45.479 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:14:45.479 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:14:45.479 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:14:45.479 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:14:45.480 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:14:45.480 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:45.480 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:14:45.480 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:14:45.480 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:14:45.481 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:14:45.481 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:14:45.481 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:14:45.481 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:14:45.481 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:45.482 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:14:45.482 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:14:45.482 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:14:45.482 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:45.482 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:14:45.483 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.483 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:14:45.483 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:14:45.483 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:14:45.484 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:14:45.484 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:14:45.484 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:14:45.484 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:14:45.484 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:14:45.485 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:14:45.485 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:14:45.485 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:14:45.485 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.485 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:14:45.486 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:14:45.486 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:14:45.486 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:14:45.486 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:14:45.486 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:45.487 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:14:45.487 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:14:45.487 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:14:45.487 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:45.487 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:14:45.488 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.488 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:14:45.488 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:14:45.488 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:14:45.488 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:14:45.489 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:14:45.489 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:14:45.489 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:14:45.489 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:14:45.489 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:14:45.490 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T13:13:40.575123Z", 2021-09-06T13:14:45.490 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:14:45.490 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.490 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.490 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:14:45.491 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:14:45.491 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:14:45.491 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:14:45.491 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:14:45.491 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:14:45.492 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:14:45.493 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T13:14:45.493 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T13:14:45.493 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:45.494 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:45.494 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:45.553 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:45.553 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:45.623 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:45.623 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:45.624 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:45.624 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:45.624 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:45.624 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:45.625 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.625 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.625 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:45.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.625 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:45.626 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.626 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.626 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:45.626 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.627 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:45.627 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.627 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:45.627 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.628 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.628 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.628 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.628 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:45.628 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.629 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.629 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:45.629 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.629 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.629 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.630 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:45.630 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.630 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.630 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.631 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:45.631 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.631 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.631 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:45.632 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.632 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.632 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.632 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:45.632 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.633 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.633 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.633 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:45.633 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:45.633 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:45.634 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:45.634 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:14:45.634 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:45.634 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:45.634 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:45.635 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:14:45.635 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:45.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.635 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.636 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:45.636 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:45.636 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:45.636 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:45.637 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:45.637 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:45.637 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:45.638 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.474951_171.1 2021-09-06T13:14:50.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:50.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:50.640 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:50.691 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:50.691 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:50.764 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:50.768 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:50.768 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:50.768 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:50.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:50.768 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:50.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:50.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:50.769 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:50.770 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:50.770 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:50.770 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:50.770 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:50.770 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:50.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:50.771 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:50.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:50.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:50.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:50.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:50.772 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:50.772 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:50.772 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:50.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:50.774 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:50.775 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:14:50.775 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:50.775 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:50.775 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:50.775 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:14:50.776 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:50.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.776 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:50.777 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:50.779 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:14:55.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:55.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:14:55.781 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:14:55.831 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:14:55.831 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:14:55.904 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:14:55.905 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:14:55.905 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:14:55.905 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:14:55.905 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:14:55.906 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:14:55.906 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:55.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:55.906 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:14:55.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:55.907 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:14:55.907 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:55.907 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:55.908 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:14:55.908 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:55.908 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:14:55.908 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:55.909 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:14:55.909 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:55.909 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:55.909 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:55.910 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:14:55.910 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:55.910 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:55.910 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:14:55.911 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:55.911 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:55.911 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:55.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:14:55.911 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:55.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:55.912 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:55.913 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:14:55.913 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:55.913 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:55.913 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:14:55.913 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:55.914 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:55.914 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:55.914 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:14:55.914 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:55.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.915 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:55.915 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:14:55.915 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:14:55.915 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:14:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:14:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:14:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:14:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:14:55.916 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:14:55.917 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:14:55.917 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:14:55.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.917 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.918 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:14:55.918 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:14:55.918 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:14:55.918 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:14:55.918 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:14:55.919 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:14:55.919 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:14:55.920 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:15:00.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:00.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:00.922 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:01.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:01.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:01.317 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:01.318 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:01.318 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:01.318 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:01.318 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:01.318 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:01.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:01.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:01.319 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:01.319 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:01.319 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:01.320 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:01.320 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:01.320 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:01.320 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:01.321 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:01.321 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:01.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:01.321 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:01.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:01.322 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:01.322 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:01.322 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:01.322 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:01.323 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:01.323 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:01.323 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:01.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:01.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:01.325 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:01.325 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:01.325 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:01.325 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:01.325 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:01.326 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:01.326 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:01.326 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:01.326 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:01.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:01.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:01.327 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:01.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:01.328 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:01.328 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:15:01.328 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:01.328 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:01.328 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:01.329 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:15:01.329 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:01.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.329 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:01.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:01.330 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:01.330 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:01.330 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:01.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:01.331 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:01.332 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:15:06.333 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:06.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:06.334 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:06.400 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:06.400 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:06.734 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:06.735 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:06.735 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:06.735 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:06.735 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:06.736 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:06.736 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:06.736 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:06.736 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:06.737 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:06.737 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:06.737 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:06.737 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:06.737 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:06.738 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:06.738 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:06.738 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:06.738 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:06.739 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:06.739 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.739 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:06.739 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:06.739 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:06.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:06.740 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:06.740 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:06.740 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:06.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:06.741 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:06.741 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:06.741 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:06.741 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:06.742 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:06.742 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:06.742 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:06.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:06.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:06.743 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:06.743 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:06.743 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.744 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:06.745 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.716832_185.1", 2021-09-06T13:15:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:06.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.716832Z", 2021-09-06T13:15:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.747 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:06.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:06.749 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:06.749 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.716832_185.1 2021-09-06T13:15:11.750 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:11.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:11.752 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:11.842 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:11.842 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:12.138 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:12.139 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:12.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:12.141 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:12.141 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:12.141 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:12.141 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:12.142 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:12.144 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:12.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:12.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:12.147 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:12.148 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:12.149 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:15:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:12.150 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:15:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.151 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.152 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:12.152 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:12.152 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:12.152 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:12.153 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:12.153 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:12.153 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:12.154 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:15:17.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:17.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:17.157 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:17.207 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:17.281 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:17.281 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:17.281 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:17.282 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:17.282 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:17.282 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:17.282 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:17.283 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:17.283 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:17.283 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:17.283 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:17.284 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:17.284 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:17.284 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:17.284 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:17.284 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:17.285 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:17.285 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:17.285 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:17.285 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.286 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:17.286 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:17.286 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:17.286 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:17.286 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:17.287 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:17.287 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:17.287 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:17.288 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:17.288 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:17.288 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:17.288 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.289 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:17.289 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:17.289 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:17.289 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:17.289 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:17.290 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:17.290 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:17.290 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:17.290 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:17.291 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:17.291 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:17.291 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.291 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:17.291 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:17.292 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:17.292 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:17.292 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:17.292 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:15:17.293 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:17.293 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:17.293 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:17.293 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:15:17.293 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:17.294 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.294 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.294 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:17.294 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:17.295 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:17.295 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:17.295 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:17.295 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:17.295 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:17.297 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:15:22.298 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:22.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:22.300 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:22.355 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:22.356 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:22.424 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:22.425 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:22.425 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:22.425 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:22.425 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:22.426 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:22.426 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:22.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:22.426 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:22.427 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:22.427 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:22.427 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:22.427 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:22.428 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:22.428 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:22.428 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:22.429 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:22.429 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:22.429 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:22.429 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:22.430 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:22.430 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:22.430 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:22.430 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:22.431 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:22.431 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:22.431 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:22.431 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:22.432 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:22.432 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:22.432 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:22.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:22.433 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:22.433 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:22.433 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:22.434 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:22.434 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:22.434 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:22.434 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:22.435 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:22.435 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:22.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:22.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:22.436 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:22.436 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:22.436 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:22.437 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:15:22.437 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:22.437 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:22.437 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:22.437 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:15:22.438 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:22.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.438 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.439 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:22.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:22.439 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:22.439 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:22.440 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:22.440 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:22.440 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:22.441 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:15:27.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:27.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:27.444 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:27.489 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:27.490 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:27.556 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:27.556 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:27.557 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:27.558 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:27.559 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.560 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:27.561 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:27.562 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:27.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:27.564 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:27.566 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:15:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:27.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:15:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:27.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:27.569 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:27.569 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:27.571 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.516029_175.1 2021-09-06T13:15:32.572 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:32.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:32.574 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:32.651 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:32.651 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:33.131 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:33.132 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:33.133 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:33.134 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:33.135 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:33.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:33.137 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:33.137 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:33.138 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:33.138 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:33.138 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:33.138 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:33.138 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:33.139 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:33.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:33.139 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.139 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:33.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:33.140 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:33.140 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:33.140 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:33.141 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:33.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:33.141 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:33.141 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:33.141 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:33.142 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:33.142 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.142 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:33.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:33.144 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:33.144 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:33.144 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:33.144 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:15:33.145 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:33.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:33.145 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:33.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:15:33.145 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:33.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.146 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.146 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:33.147 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:33.147 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:33.147 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:33.147 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:33.147 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:33.148 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:33.149 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:15:38.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:38.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:38.152 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:38.219 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:38.220 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:38.295 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:38.295 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:38.295 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:38.296 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:38.296 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:38.296 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:38.296 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:38.296 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:38.297 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:38.297 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:38.297 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:38.297 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:38.298 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:38.298 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:38.298 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:38.298 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:38.298 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:38.299 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:38.299 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:38.299 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.299 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:38.299 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:38.300 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:38.300 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:38.300 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:38.300 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:38.300 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:38.301 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:38.301 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:38.301 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:38.301 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:38.302 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.302 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:38.302 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:38.302 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:38.302 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:38.303 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:38.303 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:38.303 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:38.303 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:38.303 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:38.304 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:38.304 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:38.304 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:38.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:38.305 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:38.305 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:38.305 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:38.305 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:15:38.306 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:38.306 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:38.306 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:38.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:15:38.306 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:38.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.307 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:38.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:38.308 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:38.308 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:38.308 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:38.308 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:38.308 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:38.309 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:15:43.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:43.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:43.311 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:43.407 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:43.407 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:43.691 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:43.692 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:43.692 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:43.692 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:43.693 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:43.693 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:43.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:43.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:43.694 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:43.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:43.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:43.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:43.695 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:43.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:43.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:43.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:43.696 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:43.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:43.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:43.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:43.697 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:43.697 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:43.697 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:43.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:43.698 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:43.698 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:43.698 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:43.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:43.699 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:43.699 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:43.699 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:43.700 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:43.700 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:43.700 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:43.700 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:43.701 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:43.701 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:43.701 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:43.701 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:43.701 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:43.702 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:43.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:43.702 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:43.703 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:43.703 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:43.703 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:43.703 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:15:43.704 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:43.704 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:43.704 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:43.704 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:15:43.704 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:43.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.705 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.705 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:43.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:43.706 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:43.706 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:43.706 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:43.707 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:43.707 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:43.708 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:15:48.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:48.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:48.712 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:48.789 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:48.790 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:49.359 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:49.360 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:49.360 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:49.360 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:49.360 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:49.360 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:49.361 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:49.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:49.361 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:49.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:49.362 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:49.362 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:49.362 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:49.363 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:49.364 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:49.364 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.364 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:49.364 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:49.365 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:49.365 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:49.365 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:49.365 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:49.366 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:49.367 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:49.368 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:49.368 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:49.368 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:49.368 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:49.368 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:49.369 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:49.369 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:49.369 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:49.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:49.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:49.370 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:49.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:49.370 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:49.371 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:15:49.371 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:49.371 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:49.371 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:49.371 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:15:49.372 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:49.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.372 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:49.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:49.373 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:49.373 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:49.373 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:49.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:49.374 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:49.374 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:15:54.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:54.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:54.378 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:54.428 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:54.428 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:54.504 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:54.504 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:54.505 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:54.506 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:54.507 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:54.508 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:54.509 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:54.510 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.511 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:54.512 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:54.513 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:54.514 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:54.515 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:15:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:54.516 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:15:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:54.517 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:54.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:54.519 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:54.520 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.516029_175.1 2021-09-06T13:15:59.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:59.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:15:59.523 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:15:59.573 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:15:59.574 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:15:59.646 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:15:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:15:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:15:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:15:59.647 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:15:59.648 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:15:59.648 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:59.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:59.648 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:15:59.649 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:59.649 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:15:59.649 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:59.649 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:59.649 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:15:59.650 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:59.650 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:15:59.650 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:59.650 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:15:59.651 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:59.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:59.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:59.652 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:15:59.652 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:59.652 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:59.652 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:15:59.652 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:59.653 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:59.653 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:59.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:15:59.653 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:59.654 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:59.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:59.654 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:15:59.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:59.655 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:59.655 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:15:59.655 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:59.655 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:59.656 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:59.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:15:59.656 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:59.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:59.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:15:59.657 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:15:59.657 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:15:59.657 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:15:59.657 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:15:59.658 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:15:59.658 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:15:59.658 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:15:59.658 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:15:59.658 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:15:59.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.659 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:15:59.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:15:59.659 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:15:59.660 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:15:59.660 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:15:59.660 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:15:59.660 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:15:59.662 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.718130_186.1 2021-09-06T13:16:04.663 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:04.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:04.664 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:04.709 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:04.709 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:04.774 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:04.775 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:04.775 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:04.775 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:04.776 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:04.776 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:04.776 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:04.776 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:04.776 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:04.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:04.777 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:04.777 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:04.777 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:04.778 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:04.778 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:04.778 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:04.778 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:04.778 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:04.779 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:04.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.779 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:04.779 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:04.780 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:04.780 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:04.780 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:04.780 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:04.780 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:04.781 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:04.781 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:04.781 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:04.781 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:04.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:04.782 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:04.782 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:04.782 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:04.783 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:04.783 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:04.783 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:04.783 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:04.783 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:04.784 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:04.784 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:04.784 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.784 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:04.784 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:04.785 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:04.785 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:04.785 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:04.785 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:16:04.786 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:04.786 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:04.786 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:04.786 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:16:04.786 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:04.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.787 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.787 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:04.787 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:04.788 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:04.788 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:04.788 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:04.788 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:04.788 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:04.789 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.718130_186.1 2021-09-06T13:16:09.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:09.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:09.794 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:09.877 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:09.878 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:10.357 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:10.359 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:10.360 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:10.361 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:10.362 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:10.363 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:10.364 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:10.365 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:10.366 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:10.367 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:10.368 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.369 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:10.370 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:10.371 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:10.373 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.867618_202.1", 2021-09-06T13:16:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:10.374 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.867618Z", 2021-09-06T13:16:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.375 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:10.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:10.377 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:10.378 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.867618_202.1 2021-09-06T13:16:15.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:15.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:15.380 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:15.472 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:15.473 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:15.699 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:15.699 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:15.700 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:15.700 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:15.700 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:15.700 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:15.700 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:15.701 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:15.701 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:15.701 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:15.701 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:15.702 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:15.702 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:15.702 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:15.702 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:15.703 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:15.703 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:15.703 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:15.703 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:15.703 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.704 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:15.704 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:15.704 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:15.705 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:15.705 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:15.705 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:15.705 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:15.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.707 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:15.707 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:15.707 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:15.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:15.707 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:15.708 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:15.708 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:15.708 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:15.708 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:15.709 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:15.709 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:15.709 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:15.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:15.710 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:15.710 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:15.710 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:15.710 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:16:15.711 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:15.711 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:15.711 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:15.711 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:16:15.711 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:15.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.712 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.712 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:15.712 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:15.712 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:15.713 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:15.713 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:15.713 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:15.713 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:15.715 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:16:20.716 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:20.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:20.717 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:20.789 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:20.790 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:21.392 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:21.392 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:21.393 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:21.394 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:21.395 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.396 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:21.397 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:21.398 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:21.399 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:21.400 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:21.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:21.402 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.773406_193.1", 2021-09-06T13:16:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:21.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.773406Z", 2021-09-06T13:16:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:21.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:21.405 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:21.406 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:21.406 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.773406_193.1 2021-09-06T13:16:26.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:26.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:26.408 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:26.458 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:26.459 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:26.534 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:26.535 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:26.535 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:26.535 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:26.536 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:26.536 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:26.536 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:26.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:26.536 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:26.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:26.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:26.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:26.537 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:26.538 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:26.538 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:26.538 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:26.538 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:26.539 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:26.539 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:26.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:26.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:26.540 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:26.540 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:26.540 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:26.540 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:26.541 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:26.541 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:26.541 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:26.541 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:26.542 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:26.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.542 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:26.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:26.542 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:26.543 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:26.543 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:26.543 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:26.544 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:26.544 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:26.544 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:26.544 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:26.545 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:26.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:26.545 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:26.546 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:26.546 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:26.546 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:26.546 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:16:26.547 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:26.547 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:26.547 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:26.547 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:16:26.548 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:26.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.548 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:26.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:26.549 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:26.549 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:26.549 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:26.549 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:26.549 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:26.551 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:16:31.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:31.553 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:31.554 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:31.664 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:31.664 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:31.737 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:31.738 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:31.738 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:31.739 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:31.739 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:31.739 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:31.739 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:31.740 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:31.740 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:31.740 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:31.740 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:31.740 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:31.741 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:31.741 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:31.741 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:31.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:31.741 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:31.742 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:31.742 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:31.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.742 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:31.743 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:31.743 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:31.743 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:31.743 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:31.743 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:31.744 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:31.744 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:31.744 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:31.744 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:31.745 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:31.745 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.745 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:31.745 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:31.745 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:31.746 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:31.746 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:31.746 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:31.747 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:31.747 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:31.747 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:31.747 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:31.748 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:31.748 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.748 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:31.748 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:31.748 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:31.749 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:31.749 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:31.749 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:16:31.749 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:31.749 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:31.750 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:31.750 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:16:31.750 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:31.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.750 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.751 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:31.751 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:31.751 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:31.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:31.752 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:31.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:31.752 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:31.753 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:16:36.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:36.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:36.756 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:36.845 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:36.845 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:37.138 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:37.138 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:37.139 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:37.139 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:37.139 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:37.139 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:37.140 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:37.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:37.140 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:37.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:37.141 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:37.141 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:37.141 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:37.141 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:37.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:37.142 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:37.142 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:37.142 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:37.142 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:37.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:37.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:37.143 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:37.143 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:37.144 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:37.144 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:37.144 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:37.144 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:37.144 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:37.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:37.145 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:37.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.145 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:37.145 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:37.146 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:37.146 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:37.146 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:37.146 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:37.147 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:37.147 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:37.147 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:37.147 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:37.147 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:37.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.148 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:37.148 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:37.148 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:37.148 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:37.149 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:37.150 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:16:37.150 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:37.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.151 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:37.151 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:37.151 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:37.151 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:37.151 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:37.152 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:37.152 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:37.153 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:16:42.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:42.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:42.156 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:42.201 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:42.318 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:42.319 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:42.319 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:42.319 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:42.319 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:42.320 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:42.320 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:42.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:42.320 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:42.321 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:42.321 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:42.321 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:42.321 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:42.322 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:42.322 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:42.322 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:42.323 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:42.323 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:42.323 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:42.323 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.324 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:42.324 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:42.324 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:42.324 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:42.325 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:42.325 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:42.325 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:42.325 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:42.326 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:42.326 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:42.326 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:42.326 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.327 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:42.327 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:42.327 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:42.327 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:42.328 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:42.328 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:42.328 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:42.328 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:42.329 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.330 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:42.330 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:42.330 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:42.330 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:42.331 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:42.331 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:16:42.331 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:42.331 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:42.332 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:42.332 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:16:42.332 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:42.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.333 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.333 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:42.333 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:42.333 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:42.334 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:42.334 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:42.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:42.334 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:42.336 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:16:47.337 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:47.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:47.338 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:47.389 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:47.389 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:47.557 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:47.558 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:47.558 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:47.558 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:47.558 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:47.559 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:47.559 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:47.559 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:47.559 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:47.560 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:47.560 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:47.560 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:47.561 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:47.561 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:47.561 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:47.561 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:47.561 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:47.562 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:47.562 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:47.562 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.562 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:47.563 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:47.563 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:47.563 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:47.563 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:47.564 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:47.564 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:47.564 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:47.564 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:47.565 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:47.565 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:47.565 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.565 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:47.565 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:47.566 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:47.566 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:47.566 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:47.566 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:47.567 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:47.567 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:47.567 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:47.567 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:47.568 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:47.568 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.568 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:47.568 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:47.568 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:47.569 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:47.569 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:47.569 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:16:47.569 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:16:47.570 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:47.571 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:47.572 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:47.573 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:47.574 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:16:52.575 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:52.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:52.576 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:52.627 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:52.628 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:52.701 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:52.701 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:52.702 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:52.703 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:52.704 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:52.705 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:52.706 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:52.707 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:52.708 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:52.709 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:52.710 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:52.711 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:16:52.712 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:16:52.713 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:52.714 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:52.715 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:52.715 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:52.716 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.474951_171.1 2021-09-06T13:16:57.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:57.719 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:16:57.720 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:16:57.766 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:16:57.767 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:16:57.837 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:16:57.837 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:16:57.837 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:16:57.837 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:16:57.838 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:16:57.838 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:16:57.838 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:57.838 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:57.839 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:16:57.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:57.839 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:16:57.839 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:57.839 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:57.840 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:16:57.840 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:57.840 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:16:57.840 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:16:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:57.841 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:16:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:57.842 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:16:57.843 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:57.843 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:57.843 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:57.843 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:16:57.843 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:57.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:57.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:57.844 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:16:57.845 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:57.845 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:57.845 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:16:57.845 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:57.845 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:57.846 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:57.846 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:16:57.846 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:57.846 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:57.847 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:16:57.847 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:16:57.847 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:16:57.847 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:16:57.848 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:16:57.848 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:16:57.848 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:16:57.848 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:16:57.849 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:16:57.849 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:16:57.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.849 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.849 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:16:57.850 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:16:57.850 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:16:57.850 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:16:57.850 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:16:57.851 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:16:57.851 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:16:57.852 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:17:02.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:02.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:02.855 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:02.944 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:02.945 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:03.158 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:03.159 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:03.160 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:03.161 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:03.162 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:03.163 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:03.164 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:03.165 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:03.166 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:03.167 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.168 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:03.169 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:17:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:03.170 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:17:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.171 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:03.172 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:03.173 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:03.174 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.474951_171.1 2021-09-06T13:17:08.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:08.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:08.177 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:08.255 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:08.255 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:08.370 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:08.370 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:08.370 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:08.371 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:08.371 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:08.371 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:08.371 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:08.372 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:08.372 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:08.372 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:08.372 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:08.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:08.373 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:08.373 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:08.373 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:08.373 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:08.374 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:08.374 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:08.374 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:08.374 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:08.375 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:08.375 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:08.375 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:08.376 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:08.376 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:08.376 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:08.376 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:08.376 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:08.377 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:08.377 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:08.377 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:08.378 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:08.378 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:08.378 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:08.378 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:08.378 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:08.379 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:08.379 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:08.379 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:08.379 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:08.380 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:17:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:08.381 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:17:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:08.384 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:08.385 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:17:13.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:13.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:13.388 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:13.494 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:14.025 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:14.026 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:14.028 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:14.029 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:14.030 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:14.031 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:14.032 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:14.033 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.034 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:14.035 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:14.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:14.037 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:14.038 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:17:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:14.039 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:17:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.040 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:14.041 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:14.042 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:14.043 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:17:19.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:19.045 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:19.046 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:19.102 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:19.103 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:19.598 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:19.598 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:19.599 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:19.599 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:19.599 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:19.599 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:19.600 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:19.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:19.600 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:19.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:19.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:19.601 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:19.601 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:19.601 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:19.601 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:19.602 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:19.602 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:19.602 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:19.602 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:19.603 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.603 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:19.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:19.603 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:19.604 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:19.604 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:19.604 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:19.604 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:19.605 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:19.605 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:19.605 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:19.605 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:19.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.606 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:19.606 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:19.606 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:19.607 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:19.607 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:19.607 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:19.607 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:19.607 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:19.608 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:19.608 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:19.608 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:19.608 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.609 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:19.609 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:19.609 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:19.609 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:19.610 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:19.610 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.545985_179.1", 2021-09-06T13:17:19.610 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:19.611 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:19.611 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:19.611 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.545985Z", 2021-09-06T13:17:19.611 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:19.612 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.612 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.612 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:19.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:19.612 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:19.613 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:19.613 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:19.613 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:19.613 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:19.615 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.545985_179.1 2021-09-06T13:17:24.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:24.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:24.617 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:24.667 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:24.668 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:24.741 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:24.742 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:24.742 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:24.742 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:24.742 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:24.743 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:24.743 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:24.743 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:24.743 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:24.744 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:24.744 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:24.744 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:24.745 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:24.745 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:24.745 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:24.745 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:24.746 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:24.746 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:24.746 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:24.746 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.747 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:24.747 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:24.747 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:24.747 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:24.748 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:24.748 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:24.748 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:24.748 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:24.748 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:24.749 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:24.749 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:24.749 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.749 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:24.750 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:24.750 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:24.750 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:24.750 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:24.751 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:24.751 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:24.751 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:24.751 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:24.752 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:24.752 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:24.752 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.753 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:24.753 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:24.753 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:24.753 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:24.754 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:24.754 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.545985_179.1", 2021-09-06T13:17:24.754 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:24.754 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:24.755 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:24.755 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.545985Z", 2021-09-06T13:17:24.755 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:24.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.755 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.756 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:24.756 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:24.756 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:24.756 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:24.756 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:24.757 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:24.757 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:24.758 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.545985_179.1 2021-09-06T13:17:29.762 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:29.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:29.763 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:29.814 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:29.814 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:29.895 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:29.895 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:29.896 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:29.896 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:29.896 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:29.896 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:29.897 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:29.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:29.897 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:29.897 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:29.897 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:29.898 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:29.898 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:29.898 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:29.899 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:29.899 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:29.899 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:29.899 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:29.900 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:29.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:29.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:29.900 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:29.901 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:29.901 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:29.901 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:29.901 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:29.901 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:29.902 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:29.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:29.902 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:29.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:29.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:29.903 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:29.903 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:29.904 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:29.905 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:29.905 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:29.905 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:29.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:29.906 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:29.907 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:17:29.907 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:29.907 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:29.907 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:29.908 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:17:29.908 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:29.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.908 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.909 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:29.909 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:29.909 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:29.909 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:29.910 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:29.910 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:29.910 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:29.911 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:17:34.913 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:34.914 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:34.914 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:34.985 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:34.985 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:35.483 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:35.484 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:35.484 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:35.484 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:35.485 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:35.485 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:35.485 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:35.485 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:35.485 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:35.486 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:35.486 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:35.486 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:35.486 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:35.487 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:35.487 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:35.487 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:35.487 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:35.488 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:35.488 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:35.488 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.488 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:35.488 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:35.489 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:35.489 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:35.489 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:35.489 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:35.490 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:35.490 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:35.490 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:35.490 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:35.490 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:35.491 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.491 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:35.491 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:35.491 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:35.492 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:35.492 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:35.492 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:35.492 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:35.493 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:35.493 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:35.493 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:35.493 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:35.493 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.494 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:35.494 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:35.494 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:35.494 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:35.495 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:35.495 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:17:35.495 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:35.495 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:35.496 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:35.496 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:17:35.496 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:35.496 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.496 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.497 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:35.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:35.497 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:35.497 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:35.498 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:35.498 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:35.498 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:35.499 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:17:40.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:40.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:40.502 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:40.588 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:40.588 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:40.757 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:40.757 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:40.757 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:40.758 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:40.758 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:40.758 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:40.758 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:40.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:40.759 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:40.759 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:40.759 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:40.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:40.760 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:40.760 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:40.760 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:40.760 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:40.761 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:40.761 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:40.761 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:40.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.762 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:40.762 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:40.762 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:40.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:40.763 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:40.763 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:40.763 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:40.764 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:40.764 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:40.764 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:40.764 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:40.765 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.765 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:40.765 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:40.765 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:40.766 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:40.766 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:40.766 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:40.766 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:40.766 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:40.767 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:40.767 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:40.767 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:40.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:40.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:40.768 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:40.768 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:40.769 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:40.769 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:17:40.769 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:40.769 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:40.769 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:40.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:17:40.770 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:40.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.770 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.771 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:40.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:40.771 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:40.772 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:40.772 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:40.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:40.772 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:40.774 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.542715_178.1 2021-09-06T13:17:45.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:45.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:45.776 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:45.851 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:45.851 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:46.522 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:46.523 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:46.523 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:46.523 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:46.524 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:46.524 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:46.524 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:46.524 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:46.525 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:46.525 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:46.525 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:46.525 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:46.526 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:46.526 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:46.526 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:46.526 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:46.526 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:46.527 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:46.527 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:46.527 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.527 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:46.528 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:46.528 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:46.528 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:46.528 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:46.528 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:46.529 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:46.529 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:46.529 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:46.529 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:46.530 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:46.530 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:46.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:46.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:46.531 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:46.531 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:46.531 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:46.531 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:46.532 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:46.532 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:46.532 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:46.532 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:46.533 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:46.534 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:46.534 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:17:46.534 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:46.534 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:46.535 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:46.535 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:17:46.535 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:46.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.536 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:46.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:46.536 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:46.536 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:46.537 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:46.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:46.537 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:46.538 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.542715_178.1 2021-09-06T13:17:51.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:51.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:51.541 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:51.615 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:51.616 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:52.252 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:52.252 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:52.253 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:52.254 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:52.255 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:52.256 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:52.257 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:52.258 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:52.259 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:52.261 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:52.262 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:52.264 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:17:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:52.265 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:17:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.266 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:52.267 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:52.268 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:52.268 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:17:57.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:57.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:17:57.271 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:17:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:17:57.321 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:17:57.397 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:17:57.397 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:17:57.397 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:17:57.398 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:17:57.398 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:17:57.398 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:17:57.398 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:57.399 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:57.399 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:17:57.399 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:57.399 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:17:57.400 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:57.400 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:57.400 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:17:57.400 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:57.401 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:17:57.401 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:57.401 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:17:57.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:57.401 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.402 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:57.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:57.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:17:57.402 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:57.403 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:57.403 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:17:57.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:57.403 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:57.404 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:57.404 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:17:57.404 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:57.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:17:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:57.405 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:17:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:57.406 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:17:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:57.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:17:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:17:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:17:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:17:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:17:57.408 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:17:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:17:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:17:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:17:57.409 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:17:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:17:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:17:57.410 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:17:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:17:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:17:57.411 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:17:57.411 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:17:57.412 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:18:02.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:02.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:02.416 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:02.461 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:02.461 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:02.528 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:02.528 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:02.529 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:02.530 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:02.531 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:02.533 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:02.534 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:02.535 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:02.536 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:02.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:02.538 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:18:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:02.539 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:02.540 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:18:02.540 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:02.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:02.541 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:02.542 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:02.542 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:02.542 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:02.543 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:18:07.544 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:07.545 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:07.546 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:07.625 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:07.625 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:07.764 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:07.765 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:07.765 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:07.765 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:07.766 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:07.766 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:07.766 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:07.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:07.767 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:07.768 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:07.768 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:07.768 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:07.768 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:07.769 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:07.769 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:07.769 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:07.770 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:07.770 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:07.770 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:07.770 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:07.771 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:07.771 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:07.771 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:07.771 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:07.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:07.772 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:07.772 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:07.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:07.773 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:07.773 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:07.773 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:07.773 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:07.773 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:07.774 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:07.774 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:07.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:07.774 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:07.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:07.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:07.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:07.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:07.776 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:07.776 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:18:07.776 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:07.776 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:07.776 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:07.777 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:18:07.777 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:07.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.778 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.778 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:07.778 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:07.779 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:07.779 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:07.779 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:07.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:07.780 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:07.781 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:18:12.782 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:12.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:12.783 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:12.827 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:12.827 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:12.896 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:12.897 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:12.898 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:12.899 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:12.900 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:12.901 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:12.902 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:12.903 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.904 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:12.905 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:12.906 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:12.907 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:12.908 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:18:12.909 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:12.910 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:18:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:12.911 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:12.912 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:12.913 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:12.914 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:12.914 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:12.915 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:18:17.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:17.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:17.918 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:17.994 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:17.995 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:18.526 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:18.527 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:18.528 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:18.529 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:18.530 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:18.530 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:18.531 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:18.531 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:18.531 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:18.531 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.532 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:18.532 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:18.532 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:18.532 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:18.533 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:18.533 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:18.533 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:18.533 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:18.533 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:18.534 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:18.534 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:18.534 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.534 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:18.535 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:18.535 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:18.535 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:18.535 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:18.535 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:18.536 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:18.536 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:18.536 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:18.536 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:18.536 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:18.537 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.537 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:18.537 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:18.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:18.538 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:18.538 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:18.538 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:18:18.538 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:18.538 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:18.539 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:18.539 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:18:18.539 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:18.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.540 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.540 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:18.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:18.540 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:18.540 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:18.541 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:18.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:18.541 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:18.541 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:18:23.543 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:23.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:23.544 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:23.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:23.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:23.671 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:23.671 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:23.671 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:23.671 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:23.672 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:23.672 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:23.672 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:23.672 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:23.673 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:23.673 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:23.673 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:23.673 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:23.673 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:23.674 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:23.674 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:23.674 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:23.674 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:23.675 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:23.675 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:23.675 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.675 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:23.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:23.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:23.676 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:23.676 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:23.676 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:23.677 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:23.678 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:23.679 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:23.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:23.681 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:23.682 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.683 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:23.684 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:23.685 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:23.686 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:18:28.687 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:28.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:28.689 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:28.735 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:28.735 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:28.804 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:28.805 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:28.806 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:28.807 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:28.808 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:28.809 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:28.810 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.811 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:28.812 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:28.812 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:28.812 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:28.812 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:28.812 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:28.813 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:28.813 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:28.813 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:28.813 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:28.813 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:28.814 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:18:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:28.815 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:18:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:28.816 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:28.817 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:28.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:28.818 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:28.819 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.599872_181.1 2021-09-06T13:18:33.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:33.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:33.822 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:33.927 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:33.928 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:34.216 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:34.216 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:34.217 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:34.218 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:34.219 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:34.220 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:34.221 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:34.222 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:34.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:34.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:34.225 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:34.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:34.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:18:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:18:34.228 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:34.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:34.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:34.230 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:34.232 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:18:39.232 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:39.233 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:39.234 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:39.277 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:39.420 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:39.421 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:39.422 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:39.423 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:39.424 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:39.425 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:39.426 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:39.427 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:39.427 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:39.427 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:39.427 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:39.428 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:39.428 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:39.428 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:39.428 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.429 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:39.429 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:39.429 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:39.430 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:39.430 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:39.430 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:39.430 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:39.431 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:39.431 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:39.431 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:39.431 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:39.431 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.432 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:39.432 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:39.432 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:39.433 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:39.433 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:39.433 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:18:39.433 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:39.434 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:39.434 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:39.434 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:18:39.434 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:39.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.435 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.435 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:39.435 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:39.435 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:39.436 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:39.436 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:39.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:39.436 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:39.437 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:18:44.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:44.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:44.440 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:44.533 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:44.533 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:45.105 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:45.105 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:45.105 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:45.106 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:45.106 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:45.106 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:45.107 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:45.108 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:45.109 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:45.110 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:45.111 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:45.112 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:45.112 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:45.112 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:45.112 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:45.112 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.113 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:45.113 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:45.113 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:45.113 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:45.114 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:45.114 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:45.114 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:45.115 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:45.115 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:45.115 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:45.115 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:45.115 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.116 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:45.116 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:45.116 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:45.117 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:45.117 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:45.117 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:18:45.117 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:18:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:45.118 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:45.119 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:45.120 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:45.120 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:45.121 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:18:50.123 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:50.124 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:50.124 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:50.190 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:50.191 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:50.753 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:50.754 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:50.754 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:50.754 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:50.755 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:50.755 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:50.755 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:50.755 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:50.755 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:50.756 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:50.756 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:50.756 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:50.756 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:50.757 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:50.757 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:50.757 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:50.757 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:50.757 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:50.758 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:50.758 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.758 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:50.758 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:50.759 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:50.759 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:50.759 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:50.759 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:50.760 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:50.760 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:50.760 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:50.760 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:50.760 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:50.761 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:50.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:50.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:50.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:50.762 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:50.762 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:50.762 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:50.762 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:50.763 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:50.763 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:50.763 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:50.763 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:50.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:50.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:50.764 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:50.764 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:18:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:50.765 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:18:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:50.766 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:50.767 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:50.767 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:50.768 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:18:55.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:55.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:18:55.770 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:18:55.841 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:18:55.841 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:18:56.469 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:18:56.469 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:18:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:18:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:18:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:18:56.470 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:18:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:18:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:56.471 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:18:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:18:56.472 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:18:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:18:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:56.473 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:18:56.474 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:18:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:56.475 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:18:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:56.476 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:18:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:56.477 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:18:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:18:56.478 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:18:56.479 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:18:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:18:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:18:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:18:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:18:56.480 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:18:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:18:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:18:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:18:56.481 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:18:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:18:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:18:56.482 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:18:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:18:56.483 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:18:56.483 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:18:56.484 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:19:01.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:01.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:01.486 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:01.537 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:01.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:01.632 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:01.632 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:01.633 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:01.633 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:01.633 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:01.633 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:01.634 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:01.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:01.634 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:01.634 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:01.634 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:01.635 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:01.635 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:01.635 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:01.635 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:01.635 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:01.636 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:01.636 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:01.636 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:01.636 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.637 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:01.637 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:01.637 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:01.637 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:01.637 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:01.638 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:01.639 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:01.639 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:01.639 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:01.639 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:01.640 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:01.640 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.640 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:01.640 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:01.640 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:01.641 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:01.641 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:01.641 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:01.641 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:01.641 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:01.642 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:01.642 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:01.642 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:01.642 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.643 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:01.643 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:01.643 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:01.643 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:01.644 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:01.644 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:19:01.644 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:01.644 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:01.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:01.645 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:19:01.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:01.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.645 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.646 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:01.646 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:01.646 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:01.646 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:01.646 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:01.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:01.647 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:01.648 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:19:06.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:06.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:06.651 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:06.698 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:06.698 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:06.767 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:06.767 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:06.767 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:06.768 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:06.768 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:06.768 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:06.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:06.769 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:06.769 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:06.769 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:06.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:06.770 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:06.770 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:06.770 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:06.770 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:06.770 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:06.771 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:06.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:06.771 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:06.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.772 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:06.772 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:06.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:06.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:06.772 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:06.773 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:06.773 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:06.773 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:06.773 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:06.774 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:06.774 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:06.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:06.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:06.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:06.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:06.775 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:06.775 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:06.776 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:06.776 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:06.776 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:06.776 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:06.777 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:06.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:06.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:06.777 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:06.778 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:06.778 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:06.778 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:19:06.778 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:06.779 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:06.779 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:06.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:19:06.779 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:06.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.780 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:06.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:06.780 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:06.781 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:06.781 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:06.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:06.781 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:06.782 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:19:11.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:11.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:11.785 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:11.862 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:11.863 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:11.981 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:11.982 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:11.983 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:11.984 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:11.985 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.986 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:11.987 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:11.988 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:11.989 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:11.990 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:11.991 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:11.992 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:19:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:11.993 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:19:11.994 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:11.995 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:11.996 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:11.996 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:11.997 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:19:16.998 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:16.999 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:17.000 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:17.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:17.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:17.189 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:17.190 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:17.190 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:17.190 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:17.191 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:17.191 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:17.191 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:17.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:17.191 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:17.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:17.192 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:17.192 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:17.192 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:17.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:17.194 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:17.194 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:17.194 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:17.195 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:17.195 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:17.195 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:17.195 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:17.195 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:17.196 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:17.196 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:17.197 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:17.197 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:17.197 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:17.197 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.198 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:17.198 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:17.198 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:17.198 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:17.198 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:17.199 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:17.199 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:17.199 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:17.199 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:17.200 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:17.200 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:17.200 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:17.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:17.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:17.201 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:17.201 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:17.201 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:19:17.202 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:17.202 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:17.202 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:17.202 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:19:17.203 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:17.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.203 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.203 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:17.204 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:17.205 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:17.205 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:17.206 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:19:22.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:22.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:22.210 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:22.287 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:22.288 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:22.892 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:22.893 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:22.894 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:22.895 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:22.896 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.897 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:22.898 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:22.898 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:22.898 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:22.898 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:22.899 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:22.899 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:22.899 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:22.899 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:22.899 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:22.900 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:22.900 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.900 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:22.900 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:22.901 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:22.901 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:22.901 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:22.901 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:22.901 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:22.902 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:22.902 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:22.902 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:22.902 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:22.903 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.903 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:22.903 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:22.903 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:19:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:22.904 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:22.905 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:22.905 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:19:22.905 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:22.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.905 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.906 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:22.906 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:22.906 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:22.906 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:22.907 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:22.907 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:22.907 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:22.907 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:19:27.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:27.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:27.910 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:27.961 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:27.961 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:28.034 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:28.034 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:28.035 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:28.036 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:28.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:28.038 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:28.039 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:28.040 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:28.040 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:28.041 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:28.042 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:28.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:28.044 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:28.045 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:28.046 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:28.047 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:28.049 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:28.050 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:28.051 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:28.052 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:28.054 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:28.055 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:19:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:28.056 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:28.057 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:19:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:28.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:28.060 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:28.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:28.061 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:28.063 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.718130_186.1 2021-09-06T13:19:33.065 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:33.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:33.067 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:33.112 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:33.112 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:33.180 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:33.180 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:33.180 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:33.181 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:33.181 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:33.181 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:33.181 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:33.182 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:33.182 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:33.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:33.182 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:33.182 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:33.183 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:33.183 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:33.183 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:33.183 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:33.184 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:33.184 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:33.184 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:33.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:33.185 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:33.185 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:33.185 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:33.185 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:33.186 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:33.186 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:33.186 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:33.186 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:33.187 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:33.187 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:33.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:33.188 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:33.189 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:33.189 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:33.190 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:33.190 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:33.190 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:33.191 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:33.191 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:33.192 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:33.193 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:33.193 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:33.193 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:33.193 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:19:33.194 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:33.194 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:33.194 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:33.194 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:19:33.194 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:33.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.195 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.195 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:33.195 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:33.196 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:33.196 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:33.196 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:33.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:33.196 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:33.198 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:19:38.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:38.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:38.201 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:38.259 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:38.260 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:38.448 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:38.449 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:38.449 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:38.449 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:38.452 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:38.452 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:38.452 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:38.452 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:38.453 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:38.453 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:38.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.453 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:38.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:38.454 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:38.454 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:38.454 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:38.454 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:38.455 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:38.455 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:38.455 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:38.455 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:38.456 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:38.456 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.456 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:38.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:38.456 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:38.457 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:38.457 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:38.457 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:38.457 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:38.458 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:38.458 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:38.458 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:38.458 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:38.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.459 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:38.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:38.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:38.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:38.460 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:38.460 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:19:38.460 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:38.460 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:38.461 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:38.461 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:19:38.461 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:38.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.462 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.462 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:38.462 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:38.462 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:38.463 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:38.463 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:38.463 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:38.463 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:38.464 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:19:43.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:43.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:43.468 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:43.556 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:43.556 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:44.177 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:44.177 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:44.177 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:44.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:44.179 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:44.179 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:44.179 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:44.179 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:44.180 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:44.180 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:44.180 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:44.180 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:44.180 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:44.181 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:44.181 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:44.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:44.182 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:44.182 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:44.182 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:44.182 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:44.182 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:44.183 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:44.183 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:44.183 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:44.183 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:44.184 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:44.184 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.184 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:44.184 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:44.184 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:44.185 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:44.185 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:44.185 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:44.185 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:44.186 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:44.186 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:44.186 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:44.186 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:44.186 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.187 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:44.187 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:44.187 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:44.187 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:44.188 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:44.188 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:19:44.188 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:44.188 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:44.188 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:44.189 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:19:44.189 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:44.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.189 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.189 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:44.190 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:44.190 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:44.190 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:44.190 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:44.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:44.191 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:44.191 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:19:49.192 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:49.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:49.194 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:49.249 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:49.249 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:49.727 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:49.728 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:49.728 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:49.728 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:49.729 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:49.729 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:49.729 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:49.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:49.729 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:49.730 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:49.730 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:49.730 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:49.730 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:49.731 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:49.731 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:49.731 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:49.731 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:49.732 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:49.732 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:49.732 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.732 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:49.732 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:49.733 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:49.733 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:49.733 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:49.733 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:49.734 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:49.734 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:49.734 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:49.734 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:49.735 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:49.735 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.735 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:49.735 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:49.735 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:49.736 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:49.736 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:49.736 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:49.736 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:49.737 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:49.737 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:49.737 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:49.737 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:49.737 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.738 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:49.738 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:49.738 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:49.738 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:49.739 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:49.739 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:19:49.739 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:49.739 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:49.739 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:49.740 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:19:49.740 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:49.740 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.740 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.741 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:49.741 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:49.741 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:49.741 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:49.741 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:49.742 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:49.742 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:49.743 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:19:54.745 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:54.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:54.747 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:54.797 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:54.798 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:19:54.874 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:19:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:19:54.874 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:19:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:19:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:19:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:19:54.875 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:19:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:19:54.876 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:19:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:54.877 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:19:54.878 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:54.878 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:19:54.878 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:54.878 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.878 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:54.879 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:54.879 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:19:54.879 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:54.879 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:54.880 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:19:54.880 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:54.880 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:54.880 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:54.880 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:19:54.881 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:54.881 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.881 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:54.881 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:54.882 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:19:54.882 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:54.882 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:54.882 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:19:54.882 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:54.883 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:54.883 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:54.883 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:19:54.883 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:54.884 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.884 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:54.884 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:19:54.884 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:19:54.884 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:19:54.885 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:19:54.885 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:19:54.885 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:19:54.885 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:19:54.885 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:19:54.886 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:19:54.886 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:19:54.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.886 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.887 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:19:54.887 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:19:54.887 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:19:54.887 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:19:54.887 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:19:54.888 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:19:54.888 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:19:54.889 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.516029_175.1 2021-09-06T13:19:59.891 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:59.892 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:19:59.892 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:19:59.953 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:19:59.954 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:00.023 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:00.024 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:00.025 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:00.026 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:00.027 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:00.027 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:00.027 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:00.027 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:00.028 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:00.029 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:00.030 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:00.030 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:00.030 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:00.030 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:00.031 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.031 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:00.031 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:00.031 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:00.031 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:00.032 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:00.032 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:00.032 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:00.032 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:00.033 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:00.033 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:00.033 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:00.033 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.033 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:00.034 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:00.034 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:00.034 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:00.034 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:00.035 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T13:20:00.035 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:00.035 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:00.035 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:00.036 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T13:20:00.036 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:00.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.036 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.036 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:00.037 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:00.037 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:00.037 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:00.037 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:00.038 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:00.038 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:00.039 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.713479_184.1 2021-09-06T13:20:05.041 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... FAIL 2021-09-06T13:20:05.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.044 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:05.113 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:05.113 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:05.164 INFO:teuthology.orchestra.run.smithi041.stderr:No --period given, using current period=588058fc-8215-429d-a24d-21295d48dfdc 2021-09-06T13:20:05.185 INFO:teuthology.orchestra.run.smithi041.stdout:[ 2021-09-06T13:20:05.185 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.185 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:05.185 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:45:15.333383Z" 2021-09-06T13:20:05.186 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.186 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.186 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:20:05.186 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "0.000000" 2021-09-06T13:20:05.186 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.187 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.187 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:05.187 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T12:47:39.927055Z" 2021-09-06T13:20:05.187 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.188 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.188 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:20:05.188 INFO:teuthology.orchestra.run.smithi041.stdout: "last_update": "2021-09-06T13:13:40.575123Z" 2021-09-06T13:20:05.188 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.188 INFO:teuthology.orchestra.run.smithi041.stdout:] 2021-09-06T13:20:05.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-09-06T13:20:05.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.191 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:05.263 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:05.264 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:05.790 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:05.791 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:05.791 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:05.791 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:05.792 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:05.792 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:05.792 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:05.792 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:05.793 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:05.794 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:05.794 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:05.794 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:05.794 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:05.795 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:05.795 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:05.795 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:05.796 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.796 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:05.796 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:05.797 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:05.797 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:05.797 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:05.797 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:05.798 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:05.798 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:05.799 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:05.799 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:05.799 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:05.799 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:05.800 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:05.801 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:05.801 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:05.801 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:05.801 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:05.801 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:05.802 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:05.802 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:05.802 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.802 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:05.803 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:05.803 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:05.803 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:05.803 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:05.804 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:20:05.804 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:05.804 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:05.804 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:05.804 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T13:13:40.575123Z", 2021-09-06T13:20:05.805 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:05.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.805 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.805 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:05.805 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:05.806 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:05.806 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:05.806 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:05.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:05.807 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:05.807 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-09-06T13:20:05.808 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-09-06T13:20:05.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.809 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:05.856 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:05.857 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:05.952 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:20:05.953 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": { 2021-09-06T13:20:05.953 INFO:teuthology.orchestra.run.smithi041.stdout: "info": { 2021-09-06T13:20:05.953 INFO:teuthology.orchestra.run.smithi041.stdout: "status": "sync", 2021-09-06T13:20:05.954 INFO:teuthology.orchestra.run.smithi041.stdout: "num_shards": 4, 2021-09-06T13:20:05.954 INFO:teuthology.orchestra.run.smithi041.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:05.954 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:20:05.955 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.955 INFO:teuthology.orchestra.run.smithi041.stdout: "markers": [ 2021-09-06T13:20:05.955 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.955 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 0, 2021-09-06T13:20:05.956 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:20:05.956 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:20:05.956 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:05.956 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:20:05.956 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 1, 2021-09-06T13:20:05.957 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:20:05.957 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:05.957 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:20:05.957 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.958 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.958 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.958 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 1, 2021-09-06T13:20:05.958 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:20:05.959 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:20:05.959 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "", 2021-09-06T13:20:05.959 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:20:05.959 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:20:05.960 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:20:05.960 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "0.000000", 2021-09-06T13:20:05.960 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:20:05.961 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.961 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.962 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.962 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 2, 2021-09-06T13:20:05.963 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:20:05.963 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:20:05.963 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:05.964 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:20:05.964 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:20:05.964 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:20:05.964 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:05.965 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:20:05.965 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.965 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.965 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:20:05.966 INFO:teuthology.orchestra.run.smithi041.stdout: "key": 3, 2021-09-06T13:20:05.966 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:20:05.966 INFO:teuthology.orchestra.run.smithi041.stdout: "state": 1, 2021-09-06T13:20:05.966 INFO:teuthology.orchestra.run.smithi041.stdout: "marker": "1_1630934020.575123_295.1", 2021-09-06T13:20:05.967 INFO:teuthology.orchestra.run.smithi041.stdout: "next_step_marker": "", 2021-09-06T13:20:05.967 INFO:teuthology.orchestra.run.smithi041.stdout: "total_entries": 0, 2021-09-06T13:20:05.967 INFO:teuthology.orchestra.run.smithi041.stdout: "pos": 0, 2021-09-06T13:20:05.968 INFO:teuthology.orchestra.run.smithi041.stdout: "timestamp": "2021-09-06T13:13:40.575123Z", 2021-09-06T13:20:05.968 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:20:05.968 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.968 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.969 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:20:05.969 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:20:05.969 INFO:teuthology.orchestra.run.smithi041.stdout: "full_sync": { 2021-09-06T13:20:05.970 INFO:teuthology.orchestra.run.smithi041.stdout: "total": 1, 2021-09-06T13:20:05.970 INFO:teuthology.orchestra.run.smithi041.stdout: "complete": 1 2021-09-06T13:20:05.971 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:20:05.971 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:20:05.973 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-09-06T13:20:05.974 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-09-06T13:20:05.975 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:05.976 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:06.022 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:06.022 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:06.092 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:06.092 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:06.092 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:06.093 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:06.093 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:06.093 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:06.093 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:06.094 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:06.094 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:06.094 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:06.094 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:06.094 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:06.095 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:06.095 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:06.095 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:06.095 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:06.096 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:06.096 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:06.096 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:06.096 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:06.097 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:06.097 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:06.097 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:06.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:06.098 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:06.098 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:06.098 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:06.098 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:06.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:06.099 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:06.099 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:06.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:06.100 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:06.100 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:06.100 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:06.100 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:06.100 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:06.101 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:06.101 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:06.101 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:06.101 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:06.102 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.102 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:06.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:06.102 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:06.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:06.103 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:06.103 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:20:06.103 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:06.103 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:06.104 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:06.104 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:20:06.104 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:06.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.105 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.105 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:06.105 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:06.105 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:06.105 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:06.106 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:06.106 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:06.106 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:06.107 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.650194_182.1 2021-09-06T13:20:11.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:11.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:11.110 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:11.192 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:11.193 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:11.619 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:11.620 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:11.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:11.622 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:11.623 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:11.625 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:11.626 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:11.627 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:11.628 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:11.629 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:11.630 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:11.631 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.742338_191.1", 2021-09-06T13:20:11.632 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.742338Z", 2021-09-06T13:20:11.633 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:11.634 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:11.635 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:11.636 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:11.636 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:11.637 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.742338_191.1 2021-09-06T13:20:16.639 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:16.640 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:16.640 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:16.710 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:16.711 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:16.808 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:16.808 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:16.809 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:16.809 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:16.809 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:16.809 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:16.809 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:16.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:16.810 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:16.810 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:16.810 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:16.811 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:16.811 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:16.811 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:16.811 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:16.812 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:16.812 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:16.812 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:16.812 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:16.812 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.813 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:16.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:16.813 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:16.813 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:16.814 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:16.814 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:16.814 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:16.814 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:16.815 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:16.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:16.815 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:16.815 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.816 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:16.816 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:16.816 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:16.816 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:16.817 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:16.817 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:16.817 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:16.817 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:16.817 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:16.818 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:16.818 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:16.818 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.818 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:16.819 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:16.819 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:16.819 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:16.819 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:16.819 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:20:16.820 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:16.820 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:16.820 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:16.820 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:20:16.821 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:16.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.821 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.821 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:16.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:16.822 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:16.822 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:16.822 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:16.822 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:16.822 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:16.824 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:20:21.825 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:21.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:21.826 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:21.900 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:21.901 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:22.494 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:22.494 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:22.494 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:22.495 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:22.495 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:22.495 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:22.495 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:22.496 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:22.496 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:22.497 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:22.497 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:22.497 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:22.497 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:22.498 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:22.498 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:22.498 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:22.498 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:22.499 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:22.500 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:22.500 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:22.500 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:22.500 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:22.501 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:22.501 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:22.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:22.501 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:22.501 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:22.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.502 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:22.502 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:22.502 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:22.503 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:22.503 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:22.503 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:22.504 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:22.504 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:22.504 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:22.504 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:22.505 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:22.505 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.505 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:22.505 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:22.506 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:22.506 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:22.507 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:22.507 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:20:22.507 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:22.507 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:22.508 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:22.508 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:20:22.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:22.508 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.508 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.509 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:22.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:22.509 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:22.509 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:22.510 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:22.510 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:22.510 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:22.512 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:20:27.513 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:27.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:27.515 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:27.565 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:27.565 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:27.666 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:27.667 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:27.667 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:27.667 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:27.667 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:27.668 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:27.668 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:27.668 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:27.668 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:27.668 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:27.669 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:27.669 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:27.669 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:27.669 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:27.670 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:27.670 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:27.670 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:27.670 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:27.670 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:27.671 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.671 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:27.671 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:27.671 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:27.671 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:27.672 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:27.672 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:27.672 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:27.672 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:27.673 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:27.673 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:27.673 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:27.673 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.673 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:27.674 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:27.674 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:27.674 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:27.674 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:27.674 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:27.675 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:27.676 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:27.676 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:27.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:27.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:27.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:27.677 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:27.677 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:20:27.677 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:27.677 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:27.678 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:27.678 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:20:27.678 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:27.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.679 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.679 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:27.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:27.679 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:27.679 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:27.680 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:27.680 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:27.680 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:27.681 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:20:32.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:32.683 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:32.684 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:32.736 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:32.736 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:32.811 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:32.811 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:32.811 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:32.811 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:32.812 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:32.812 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:32.812 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:32.812 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:32.813 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:32.813 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:32.813 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:32.813 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:32.814 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:32.814 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:32.814 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:32.814 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:32.815 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:32.815 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:32.820 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:32.820 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:32.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:32.822 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:32.822 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:32.822 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:32.823 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:32.823 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:32.823 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:32.824 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:32.824 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:32.825 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:32.825 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.826 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:32.826 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:32.827 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:32.827 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:32.828 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:32.828 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:32.828 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:32.829 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:32.829 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:32.830 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:32.830 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:32.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.831 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:32.831 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:32.832 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:32.832 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:32.832 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:32.833 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:20:32.833 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:32.833 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:32.833 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:32.834 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:20:32.834 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:32.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.834 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.834 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:32.835 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:32.835 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:32.835 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:32.835 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:32.836 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:32.836 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:32.837 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:20:37.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:37.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:37.839 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:37.922 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:37.922 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:38.205 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:38.206 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:38.206 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:38.206 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:38.206 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:38.206 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:38.207 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:38.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:38.207 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:38.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:38.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:38.209 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:38.209 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:38.209 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:38.209 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:38.210 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:38.210 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:38.210 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:38.210 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:38.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.211 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:38.211 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:38.211 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:38.211 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:38.212 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:38.212 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:38.212 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:38.212 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:38.212 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:38.213 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:38.213 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:38.213 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.213 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:38.214 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:38.214 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:38.214 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:38.214 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:38.214 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:38.215 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:38.215 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:38.215 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:38.215 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:38.215 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:38.216 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.216 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:38.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:38.216 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:38.217 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:38.217 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:38.217 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.545985_179.1", 2021-09-06T13:20:38.217 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:38.217 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:38.218 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:38.218 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.545985Z", 2021-09-06T13:20:38.218 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:38.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.219 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.219 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:38.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:38.220 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:38.220 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:38.220 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:38.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:38.220 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:38.222 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.545985_179.1 2021-09-06T13:20:43.223 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:43.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:43.224 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:43.279 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:43.280 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:43.434 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:43.434 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:43.434 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:43.434 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:43.435 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:43.435 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:43.435 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:43.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:43.436 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:43.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:43.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:43.437 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:43.438 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:43.438 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:43.438 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:43.439 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:43.439 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:43.440 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:43.440 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:43.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.441 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:43.441 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:43.442 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:43.442 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:43.443 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:43.443 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:43.443 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:43.444 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:43.444 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:43.445 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:43.445 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:43.445 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:43.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:43.446 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:43.446 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:43.447 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:43.447 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:43.447 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:43.448 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:43.448 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:43.448 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:43.448 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:43.448 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:43.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:43.449 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:43.449 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:43.450 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:43.450 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:20:43.450 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:43.450 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:43.450 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:43.451 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:20:43.451 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:43.451 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.452 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:43.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:43.452 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:43.452 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:43.453 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:43.453 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:43.453 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:43.455 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:20:48.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:48.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:48.457 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:48.538 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:48.643 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:48.643 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:48.644 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:48.644 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:48.644 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:48.645 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:48.645 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:48.645 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:48.645 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:48.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:48.646 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:48.646 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:48.646 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:48.646 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:48.647 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:48.647 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:48.647 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:48.647 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:48.648 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:48.648 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.648 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:48.648 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:48.649 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:48.649 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:48.649 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:48.649 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:48.649 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:48.650 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:48.650 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:48.650 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:48.650 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:48.651 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.651 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:48.651 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:48.651 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:48.651 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:48.652 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:48.652 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:48.652 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:48.652 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:48.653 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:48.653 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:48.653 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:48.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:48.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:48.654 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:48.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:48.655 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:48.655 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.742338_191.1", 2021-09-06T13:20:48.655 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:48.655 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:48.655 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:48.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.742338Z", 2021-09-06T13:20:48.656 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:48.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.656 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:48.657 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:48.657 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:48.657 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:48.657 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:48.658 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:48.658 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:48.659 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.742338_191.1 2021-09-06T13:20:53.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:53.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:53.662 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:53.771 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:53.771 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:20:54.317 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:20:54.318 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:20:54.318 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:20:54.318 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:20:54.318 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:20:54.319 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:20:54.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:54.319 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:54.319 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:20:54.320 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:54.320 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:20:54.320 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:54.320 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:54.321 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:20:54.321 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:54.321 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:20:54.321 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:54.321 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:20:54.322 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:54.322 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.322 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:54.323 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:54.323 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:20:54.323 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:54.323 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:54.323 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:20:54.324 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:54.324 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:54.324 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:54.324 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:20:54.325 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:54.325 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.325 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:54.325 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:54.325 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:20:54.326 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:54.326 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:54.326 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:20:54.326 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:54.327 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:54.327 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:54.327 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:20:54.327 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:54.327 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.328 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:54.328 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:20:54.328 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:20:54.328 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:20:54.329 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:20:54.329 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:20:54.329 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:20:54.329 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:20:54.329 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:20:54.330 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:20:54.330 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:20:54.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.330 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.331 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:20:54.331 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:20:54.331 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:20:54.331 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:20:54.331 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:20:54.332 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:20:54.332 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:20:54.333 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.599872_181.1 2021-09-06T13:20:59.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:59.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:20:59.336 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:20:59.414 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:20:59.415 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:00.040 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:00.041 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:00.041 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:00.041 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:00.041 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:00.042 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:00.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:00.042 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:00.042 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:00.043 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:00.043 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:00.043 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:00.043 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:00.043 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:00.044 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:00.045 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:00.046 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:00.046 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:00.046 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:00.046 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:00.047 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:00.047 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:00.047 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:00.047 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:00.048 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.048 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:00.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:00.048 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:00.048 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:00.049 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:00.049 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:00.049 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:00.049 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:00.050 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:00.050 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:00.050 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:00.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:00.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:00.051 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:00.051 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:00.051 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:00.051 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:21:00.052 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:00.052 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:00.052 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:00.053 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:21:00.053 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:00.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.053 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:00.054 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:00.054 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:00.054 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:00.054 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:00.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:00.055 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:00.055 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:21:05.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:05.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:05.058 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:05.110 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:05.198 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:05.199 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:05.200 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:05.201 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:05.202 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:05.203 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:05.204 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:05.205 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:05.206 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.207 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:05.208 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:05.209 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:05.210 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:05.211 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:05.211 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:05.211 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:05.212 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:21:05.212 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:05.213 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:05.213 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:05.213 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:21:05.213 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:05.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.214 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.214 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:05.214 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:05.215 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:05.215 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:05.215 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:05.215 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:05.215 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:05.217 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:21:10.218 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:10.219 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:10.219 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:10.270 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:10.270 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:10.343 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:10.344 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:10.344 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:10.344 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:10.345 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:10.345 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:10.345 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:10.345 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:10.346 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:10.346 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:10.346 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:10.346 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:10.347 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:10.347 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:10.347 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:10.348 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:10.348 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:10.348 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:10.348 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:10.348 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.349 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:10.349 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:10.349 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:10.349 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:10.350 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:10.350 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:10.350 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:10.350 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:10.351 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:10.351 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:10.351 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:10.351 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.351 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:10.352 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:10.352 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:10.352 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:10.352 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:10.353 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:10.354 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:10.355 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T13:21:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T13:21:10.356 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:10.357 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:10.358 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:10.358 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:10.360 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.713479_184.1 2021-09-06T13:21:15.361 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:15.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:15.363 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:15.432 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:15.433 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:15.531 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:15.532 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:15.532 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:15.532 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:15.532 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:15.533 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:15.533 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:15.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:15.533 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:15.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:15.534 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:15.534 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:15.534 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:15.535 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:15.535 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:15.535 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:15.535 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:15.535 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:15.536 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:15.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:15.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:15.536 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:15.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:15.537 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:15.537 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:15.537 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:15.538 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:15.539 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:15.540 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:15.541 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:15.542 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:15.542 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:15.542 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:21:15.542 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:15.543 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:15.543 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:15.543 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:21:15.543 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:15.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.544 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:15.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:15.544 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:15.544 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:15.545 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:15.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:15.545 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:15.547 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:21:20.548 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:20.549 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:20.549 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:20.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:20.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:20.720 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:20.720 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:20.721 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:20.722 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:20.723 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:20.724 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:20.725 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:20.726 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.727 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:20.728 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:20.729 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.730 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:20.731 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:21:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:21:20.732 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:20.733 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:20.734 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:20.735 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:20.736 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.718130_186.1 2021-09-06T13:21:25.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:25.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:25.739 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:25.813 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:25.814 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:26.444 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:26.444 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:26.445 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:26.445 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:26.445 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:26.445 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:26.446 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:26.446 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:26.446 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:26.446 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:26.447 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:26.447 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:26.447 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:26.447 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:26.448 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:26.448 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:26.448 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:26.448 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:26.448 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:26.449 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:26.450 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:26.451 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:26.452 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:26.452 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:26.452 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.452 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:26.453 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:26.453 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:26.453 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:26.453 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:26.453 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:26.454 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:26.454 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:26.454 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:26.454 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:26.455 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:26.455 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:26.455 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:26.456 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:26.456 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:26.456 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:26.456 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.545985_179.1", 2021-09-06T13:21:26.456 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:26.457 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:26.457 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:26.457 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.545985Z", 2021-09-06T13:21:26.457 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:26.457 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.458 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.458 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:26.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:26.459 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:26.459 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:26.459 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:26.459 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:26.460 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:26.460 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.545985_179.1 2021-09-06T13:21:31.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:31.463 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:31.463 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:31.516 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:31.517 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:31.592 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:31.592 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:31.593 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:31.594 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:31.595 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:31.596 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:31.597 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:31.598 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:31.599 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:31.600 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:31.601 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.602 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:31.603 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:21:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:31.604 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:21:31.605 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:31.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.605 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.605 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:31.605 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:31.606 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:31.606 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:31.606 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:31.606 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:31.607 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:31.608 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:21:36.610 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:36.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:36.611 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:36.670 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:36.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:36.766 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:36.767 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:36.767 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:36.767 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:36.767 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:36.767 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:36.768 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:36.768 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:36.768 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:36.768 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:36.769 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:36.769 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:36.769 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:36.769 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:36.770 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:36.770 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:36.770 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:36.771 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:36.771 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:36.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:36.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:36.772 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:36.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:36.772 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:36.772 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:36.773 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:36.773 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:36.773 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:36.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:36.774 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:36.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:36.774 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:36.774 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:36.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:36.775 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:36.775 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:36.775 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:36.776 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:36.776 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:36.776 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:36.776 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:36.776 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:36.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:36.777 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:36.777 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:36.778 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:36.778 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:21:36.778 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:36.778 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:36.778 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:36.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:21:36.779 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:36.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.779 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.780 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:36.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:36.780 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:36.780 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:36.780 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:36.781 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:36.781 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:36.782 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:21:41.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:41.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:41.785 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:41.837 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:41.838 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:42.045 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:42.045 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:42.046 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:42.046 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:42.046 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:42.047 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:42.047 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:42.047 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:42.048 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:42.048 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:42.048 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:42.048 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:42.049 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.050 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:42.051 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:42.051 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:42.051 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:42.051 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:42.051 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:42.052 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:42.052 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:42.052 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:42.052 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:42.053 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:42.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.053 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:42.053 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:42.053 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:42.054 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:42.054 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:42.054 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:42.054 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:42.055 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:42.055 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:42.055 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:42.055 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:42.055 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:42.056 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:42.056 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:42.056 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:42.056 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:42.057 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:21:42.057 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:42.057 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:42.057 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:42.058 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:21:42.058 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:42.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.058 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.058 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:42.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:42.059 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:42.059 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:42.059 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:42.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:42.060 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:42.060 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:21:47.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:47.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:47.063 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:47.168 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:47.169 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:47.453 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:47.454 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:47.454 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:47.454 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:47.455 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:47.456 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:47.457 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.458 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:47.459 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:47.460 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:47.461 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:47.462 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:47.463 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:47.464 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:21:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:47.465 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:21:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.466 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:47.467 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:47.468 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:47.468 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:47.469 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:21:52.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:52.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:52.473 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:52.524 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:52.525 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:52.652 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:52.652 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:52.653 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:52.654 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:52.655 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.656 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:52.657 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:52.658 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:52.659 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:52.660 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:52.661 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:52.662 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:21:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:52.663 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:21:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.664 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:52.665 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:52.666 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:52.667 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:21:57.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:57.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:21:57.670 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:21:57.721 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:21:57.721 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:21:57.796 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:21:57.797 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:21:57.797 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:21:57.797 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:21:57.798 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:21:57.798 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:21:57.798 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:57.798 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:57.798 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:21:57.799 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:57.799 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:21:57.799 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:57.799 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:57.800 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:21:57.800 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:57.800 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:21:57.800 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:57.800 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:21:57.801 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:57.801 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.801 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:57.801 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:57.802 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:21:57.802 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:57.802 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:57.802 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:21:57.803 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:57.803 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:57.803 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:57.803 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:21:57.803 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:57.804 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.804 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:57.804 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:57.804 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:21:57.805 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:57.805 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:57.805 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:21:57.805 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:57.805 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:57.806 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:57.806 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:21:57.806 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:57.806 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.807 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:57.807 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:21:57.807 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:21:57.807 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:21:57.807 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:21:57.808 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:21:57.808 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:21:57.808 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:21:57.808 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:21:57.809 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:21:57.809 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:21:57.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.809 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.809 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:21:57.810 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:21:57.810 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:21:57.810 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:21:57.810 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:21:57.811 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:21:57.811 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:21:57.812 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:22:02.814 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:02.814 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:02.815 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:02.906 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:03.215 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:03.216 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:03.217 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:03.218 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:03.219 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:03.220 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:03.221 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:03.222 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.223 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:03.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:03.225 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:03.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:03.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:22:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:03.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:22:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:03.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:03.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:03.231 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:03.232 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:22:08.233 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:08.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:08.234 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:08.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:08.308 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:08.381 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:08.382 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:08.383 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:08.384 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:08.385 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:08.385 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:08.385 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:08.385 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:08.386 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:08.386 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.386 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:08.387 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:08.387 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:08.387 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:08.387 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:08.388 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:08.388 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:08.388 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:08.388 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:08.388 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:08.389 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:08.389 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.389 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:08.389 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:08.390 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:08.390 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:08.390 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:08.390 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:08.390 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:08.391 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:08.391 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:08.391 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:08.392 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:08.392 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.392 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:08.392 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:08.393 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:08.393 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:08.393 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:08.393 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.510450_174.1", 2021-09-06T13:22:08.393 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:08.394 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:08.394 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:08.394 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.510450Z", 2021-09-06T13:22:08.394 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:08.395 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.395 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.395 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:08.395 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:08.395 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:08.396 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:08.396 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:08.396 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:08.396 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:08.397 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.510450_174.1 2021-09-06T13:22:13.399 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:13.400 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:13.401 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:14.054 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:14.055 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:14.056 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:14.057 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:14.058 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:14.059 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:14.060 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:14.061 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:14.062 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:14.063 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:14.064 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:14.065 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:22:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:14.066 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:22:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.067 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:14.068 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:14.069 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:14.069 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:14.070 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:22:19.072 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:19.073 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:19.074 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:19.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:19.213 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:19.214 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:19.214 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:19.214 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:19.215 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:19.215 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:19.215 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:19.215 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:19.216 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:19.216 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:19.216 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:19.216 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:19.217 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:19.217 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:19.217 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:19.217 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:19.218 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:19.218 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:19.218 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:19.218 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.219 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:19.219 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:19.219 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:19.219 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:19.220 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:19.220 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:19.220 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:19.220 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:19.221 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:19.221 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:19.221 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:19.221 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.222 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:19.222 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:19.222 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:19.224 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:19.224 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:19.225 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:19.225 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:19.225 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:19.225 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:19.226 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:19.226 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:19.226 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:19.227 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:19.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:19.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:19.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:19.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:22:19.228 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:19.228 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:19.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:19.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:22:19.228 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:19.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.229 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:19.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:19.230 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:19.230 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:19.230 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:19.230 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:19.230 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:19.234 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.516029_175.1 2021-09-06T13:22:24.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:24.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:24.236 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:24.287 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:24.288 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:24.363 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:24.363 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:24.364 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:24.364 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:24.364 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:24.365 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:24.365 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:24.365 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:24.365 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:24.366 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:24.366 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:24.366 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:24.366 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:24.367 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:24.367 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:24.367 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:24.367 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:24.368 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:24.368 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:24.368 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.368 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:24.368 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:24.369 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:24.369 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:24.369 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:24.369 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:24.370 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:24.371 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:24.372 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:24.373 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:24.374 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.517790_176.1", 2021-09-06T13:22:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:24.375 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.517790Z", 2021-09-06T13:22:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.376 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:24.377 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:24.378 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:24.378 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:24.379 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.517790_176.1 2021-09-06T13:22:29.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:29.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:29.382 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:29.426 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:29.427 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:29.494 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:29.494 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:29.495 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:29.495 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:29.495 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:29.496 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:29.496 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:29.497 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:29.497 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:29.498 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:29.499 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:29.500 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:29.501 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:29.501 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:29.502 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:29.502 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:29.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:29.505 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:29.505 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:29.506 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:29.506 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:29.507 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:29.507 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:29.507 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:29.508 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:29.508 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:29.509 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.509 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:29.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:29.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:29.510 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:29.511 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:29.511 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:29.512 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:29.512 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:29.513 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:29.514 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:29.514 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:29.514 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.515 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:29.515 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:29.516 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:29.516 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:29.516 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:29.516 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:22:29.517 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:29.517 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:29.517 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:29.517 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:22:29.518 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:29.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.518 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.518 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:29.518 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:29.519 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:29.519 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:29.519 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:29.519 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:29.520 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:29.521 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:22:34.522 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:34.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:34.523 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:34.568 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:34.568 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:34.680 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:34.680 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:34.681 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:34.682 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:34.683 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:34.684 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:34.686 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:34.687 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:34.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:34.689 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:34.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:34.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:34.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.545985_179.1", 2021-09-06T13:22:34.693 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:34.693 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:34.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:34.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.545985Z", 2021-09-06T13:22:34.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:34.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.694 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.694 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:34.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:34.695 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:34.696 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:34.696 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:34.697 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:34.697 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:34.698 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.545985_179.1 2021-09-06T13:22:39.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:39.700 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:39.701 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:39.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:39.795 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:40.250 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:40.250 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:40.251 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:40.252 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:40.253 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:40.254 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:40.255 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:40.256 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:40.256 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:40.256 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:40.256 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:40.257 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:40.257 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:40.257 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.257 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:40.257 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:40.258 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:40.258 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:40.258 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:40.258 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:40.258 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:40.259 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:40.259 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:40.259 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:40.259 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:40.260 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:40.261 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:40.261 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:22:40.261 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:40.261 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:40.262 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:40.262 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:22:40.262 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:40.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.262 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.263 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:40.263 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:40.263 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:40.263 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:40.264 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:40.264 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:40.264 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:40.265 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:22:45.266 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:45.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:45.268 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:45.359 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:45.359 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:45.758 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:45.759 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:45.759 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:45.760 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:45.760 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:45.760 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:45.760 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:45.761 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:45.761 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:45.761 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:45.761 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:45.762 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:45.762 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:45.762 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:45.763 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:45.763 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:45.763 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:45.763 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:45.764 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:45.764 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.764 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:45.764 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:45.764 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:45.765 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:45.765 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:45.765 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:45.765 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:45.766 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:45.766 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:45.766 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:45.766 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:45.766 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.767 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:45.767 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:45.767 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:45.767 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:45.768 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:45.768 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:45.768 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:45.768 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:45.770 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:45.770 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:45.770 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:45.771 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.771 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:45.771 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:45.771 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:45.772 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:45.772 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:45.772 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:22:45.772 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:45.772 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:45.773 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:45.773 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:22:45.773 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:45.773 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.774 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.774 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:45.774 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:45.774 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:45.775 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:45.775 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:45.775 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:45.776 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:45.777 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:22:50.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:50.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:50.780 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:50.854 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:50.855 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:51.429 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:51.429 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:51.429 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:51.429 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:51.430 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:51.430 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:51.430 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:51.430 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:51.431 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:51.431 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:51.431 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:51.431 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:51.431 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:51.432 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:51.432 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:51.432 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:51.432 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:51.433 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:51.433 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:51.433 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.433 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:51.433 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:51.434 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:51.434 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:51.434 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:51.435 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:51.435 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:51.435 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:51.435 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:51.435 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:51.436 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:51.436 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.436 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:51.436 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:51.437 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:51.437 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:51.437 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:51.437 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:51.438 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:51.438 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:51.438 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:51.438 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:51.438 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:51.439 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.439 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:51.439 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:51.439 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:51.440 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:51.440 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:51.440 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:22:51.440 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:51.440 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:51.441 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:51.441 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:22:51.441 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:51.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.441 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.442 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:51.442 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:51.442 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:51.442 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:51.443 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:51.443 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:51.443 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:51.443 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.650194_182.1 2021-09-06T13:22:56.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:56.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:22:56.447 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:22:56.498 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:22:56.498 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:22:56.570 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:22:56.570 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:22:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:22:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:22:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:22:56.571 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:22:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:22:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:56.572 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:22:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:22:56.573 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:22:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:22:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:56.574 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:22:56.575 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:22:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:56.576 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:22:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:56.577 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:22:56.578 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:22:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:56.579 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:22:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.580 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:22:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:22:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:22:56.581 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:22:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.474951_171.1", 2021-09-06T13:22:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:22:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:22:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:22:56.582 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.474951Z", 2021-09-06T13:22:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:22:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:22:56.583 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:22:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:22:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:22:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:22:56.584 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:22:56.585 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:22:56.586 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.474951_171.1 2021-09-06T13:23:01.587 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:01.588 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:01.588 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:01.640 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:01.641 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:01.715 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:01.716 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:01.716 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:01.716 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:01.716 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:01.717 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:01.717 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:01.717 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:01.717 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:01.717 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:01.718 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:01.718 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:01.718 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:01.718 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:01.719 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:01.719 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:01.719 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:01.719 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:01.720 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:01.720 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.720 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:01.720 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:01.721 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:01.721 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:01.721 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:01.721 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:01.722 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:01.722 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:01.722 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:01.722 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:01.723 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:01.723 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.723 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:01.723 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:01.723 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:01.724 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:01.724 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:01.724 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:01.724 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:01.725 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:01.725 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:01.725 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:01.725 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:01.725 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.726 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:01.726 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:01.726 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:01.726 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:01.727 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:01.727 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:23:01.727 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:01.727 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:01.727 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:01.728 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:23:01.728 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:01.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.728 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.729 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:01.729 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:01.729 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:01.729 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:01.729 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:01.730 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:01.730 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:01.731 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:23:06.732 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:06.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:06.734 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:06.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:06.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:07.094 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:07.094 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:07.094 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:07.095 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:07.095 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:07.095 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:07.095 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:07.096 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:07.096 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:07.096 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:07.096 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:07.096 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:07.097 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:07.097 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:07.097 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:07.097 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:07.098 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:07.098 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:07.098 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:07.098 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.099 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:07.099 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:07.099 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:07.099 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:07.100 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:07.100 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:07.100 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:07.100 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:07.100 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:07.101 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:07.101 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:07.101 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.101 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:07.102 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:07.102 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:07.102 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:07.102 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:07.102 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:07.103 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:07.103 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:07.103 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:07.103 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:07.104 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:07.104 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.104 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:07.104 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:23:07.105 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:07.106 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:07.106 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:07.106 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:23:07.106 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:07.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.107 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.107 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:07.107 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:07.108 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:07.108 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:07.108 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:07.108 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:07.108 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:07.110 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:23:12.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:12.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:12.112 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:12.157 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:12.158 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:12.233 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:12.234 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:12.234 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:12.234 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:12.234 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:12.235 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:12.235 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:12.235 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:12.235 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:12.236 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:12.236 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:12.236 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:12.236 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:12.237 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:12.237 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:12.237 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:12.237 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:12.238 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:12.238 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:12.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.239 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:12.239 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:12.239 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:12.240 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:12.240 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:12.241 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:12.241 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:12.242 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:12.243 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:12.243 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:12.244 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:12.244 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.244 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:12.244 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:12.245 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:12.245 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:12.245 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:12.245 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:12.246 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:12.246 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:12.246 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:12.246 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:12.247 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:12.247 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.247 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:12.247 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:12.248 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:12.248 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:12.248 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:12.248 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.471383_170.1", 2021-09-06T13:23:12.249 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:12.249 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:12.249 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:12.249 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.471383Z", 2021-09-06T13:23:12.250 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:12.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.250 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.250 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:12.250 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:12.251 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:12.251 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:12.251 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:12.251 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:12.252 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:12.253 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.471383_170.1 2021-09-06T13:23:17.254 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:17.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:17.256 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:17.352 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:17.353 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:18.037 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:18.037 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:18.039 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:18.040 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:18.041 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:18.042 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:18.043 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.044 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:18.044 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:18.044 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:18.044 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:18.045 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:18.045 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:18.045 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:18.045 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:18.045 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:18.046 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:18.046 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:18.046 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.046 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:18.047 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:18.047 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:18.047 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:18.047 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:18.047 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:18.048 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:18.048 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:18.048 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:18.048 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:18.049 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:18.050 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:23:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:18.051 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:23:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:18.052 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:18.053 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:18.053 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:18.053 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:18.053 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:18.054 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:18.055 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:23:23.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:23.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:23.057 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:23.108 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:23.109 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:23.183 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:23.184 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:23.185 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:23.186 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:23.186 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:23.186 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:23.186 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:23.187 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:23.187 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:23.187 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:23.187 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:23.187 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:23.188 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:23.188 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:23.188 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.188 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:23.189 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:23.189 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:23.189 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:23.189 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:23.189 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:23.190 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:23.190 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:23.190 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:23.190 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:23.191 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:23.191 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.191 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:23.191 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:23.192 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:23.192 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:23.192 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:23.192 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:23.192 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:23.193 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:23.193 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:23.193 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:23.193 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:23.194 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.194 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:23.194 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:23.194 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:23.194 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:23.195 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:23.195 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:23:23.195 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:23.195 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:23.196 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:23.196 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:23:23.196 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:23.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.196 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.197 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:23.197 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:23.197 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:23.197 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:23.198 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:23.198 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:23.198 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:23.199 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:23:28.201 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:28.201 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:28.202 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:28.256 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:28.257 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:28.330 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:28.330 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:28.331 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:28.331 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:28.331 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:28.331 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:28.331 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:28.332 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:28.332 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:28.332 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:28.332 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:28.332 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:28.333 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:28.333 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:28.333 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:28.334 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:28.334 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:28.334 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:28.334 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:28.334 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.335 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:28.335 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:28.335 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:28.335 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:28.336 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:28.336 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:28.336 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:28.336 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:28.336 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:28.337 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:28.337 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:28.337 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.337 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:28.337 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:28.338 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:28.338 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:28.338 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:28.338 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:28.339 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:28.339 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:28.339 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:28.339 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:28.339 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:28.340 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.340 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:28.340 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:28.340 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:28.340 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:28.341 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:28.341 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:23:28.341 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:28.341 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:28.342 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:28.342 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:23:28.342 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:28.342 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.342 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.343 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:28.343 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:28.343 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:28.343 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:28.343 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:28.344 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:28.344 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:28.345 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:23:33.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:33.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:33.348 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:33.442 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:33.443 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:33.683 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:33.684 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:33.684 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:33.684 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:33.684 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:33.685 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:33.686 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:33.686 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:33.686 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:33.686 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:33.687 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:33.687 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:33.687 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:33.687 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:33.687 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:33.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.688 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:33.688 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:33.688 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:33.689 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:33.689 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:33.689 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:33.689 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:33.690 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:33.690 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:33.690 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:33.690 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:33.690 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.691 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:33.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:33.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:33.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:33.692 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:33.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:33.692 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:33.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:33.693 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:33.693 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:33.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:33.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.694 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:33.694 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:33.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:33.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:33.695 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:33.695 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.523913_177.1", 2021-09-06T13:23:33.695 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:33.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:33.695 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:33.696 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.523913Z", 2021-09-06T13:23:33.696 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:33.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.697 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:33.697 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:33.697 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:33.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:33.697 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:33.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:33.698 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:33.699 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.523913_177.1 2021-09-06T13:23:38.701 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:38.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:38.702 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:38.763 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:38.764 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:38.927 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:38.928 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:38.929 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:38.930 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:38.931 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:38.932 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:38.933 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:38.934 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.935 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:38.936 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:38.937 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:38.938 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:38.939 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.773406_193.1", 2021-09-06T13:23:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.773406Z", 2021-09-06T13:23:38.940 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:38.941 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:38.942 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:38.942 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:38.943 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.773406_193.1 2021-09-06T13:23:43.944 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:43.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:43.945 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:44.040 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:44.041 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:44.541 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:44.541 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:44.542 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:44.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:44.544 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:44.545 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:44.546 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:44.547 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:44.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:44.549 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:44.550 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.551 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:44.552 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:23:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:44.553 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:23:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.554 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:44.555 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:44.556 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:44.556 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:23:49.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:49.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:49.559 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:49.611 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:49.611 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:49.688 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:49.689 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:49.689 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:49.689 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:49.690 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:49.690 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:49.690 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:49.690 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:49.690 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:49.691 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:49.691 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:49.691 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:49.691 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:49.692 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:49.692 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:49.692 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:49.692 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:49.692 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:49.693 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:49.693 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.693 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:49.693 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:49.694 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:49.694 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:49.694 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:49.694 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:49.694 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:49.695 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:49.695 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:49.695 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:49.695 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:49.696 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.696 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:49.696 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:49.696 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:49.696 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:49.697 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:49.697 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:49.697 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:49.697 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:49.698 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:49.698 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:49.698 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:49.698 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.699 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:49.699 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:49.699 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:49.699 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:49.699 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:49.700 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.718130_186.1", 2021-09-06T13:23:49.700 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:49.700 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:49.700 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:49.700 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.718130Z", 2021-09-06T13:23:49.701 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:49.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.701 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.701 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:49.702 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:49.702 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:49.702 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:49.702 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:49.702 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:49.703 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:49.704 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.718130_186.1 2021-09-06T13:23:54.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:54.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:54.707 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:54.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:54.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:23:54.834 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:23:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:23:54.834 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:23:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:23:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:23:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:23:54.835 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:23:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:23:54.836 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:23:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:54.837 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:23:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:23:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:54.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:23:54.839 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:23:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:54.840 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:23:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.841 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:23:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:54.842 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:23:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:54.843 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:23:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:23:54.844 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:23:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:23:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:23:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.599872_181.1", 2021-09-06T13:23:54.845 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:23:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:23:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:23:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.599872Z", 2021-09-06T13:23:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:23:54.846 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:23:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:23:54.847 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:23:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:23:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:23:54.848 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:23:54.848 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:23:54.849 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.599872_181.1 2021-09-06T13:23:59.850 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:59.851 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:23:59.852 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:23:59.939 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:23:59.940 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:00.133 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:00.134 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:00.134 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:00.135 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:00.135 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:00.136 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:00.136 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:00.136 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:00.136 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:00.137 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:00.137 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:00.137 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:00.137 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:00.138 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:00.138 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:00.139 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:00.140 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:00.141 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:00.142 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:00.143 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:00.144 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:00.145 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.146 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:00.146 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:00.146 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:00.146 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:00.147 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:00.147 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.542715_178.1", 2021-09-06T13:24:00.147 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:00.147 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:00.148 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:00.148 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.542715Z", 2021-09-06T13:24:00.148 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:00.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.148 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.149 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:00.149 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:00.149 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:00.150 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:00.150 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:00.150 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:00.150 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:00.152 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.542715_178.1 2021-09-06T13:24:05.154 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:05.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:05.155 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:05.199 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:05.199 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:05.610 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:05.610 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:05.611 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:05.612 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:05.612 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:05.612 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:05.612 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:05.613 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:05.613 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:05.613 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:05.613 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:05.614 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:05.614 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:05.614 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:05.614 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:05.614 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:05.615 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.615 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:05.615 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:05.615 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:05.616 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:05.616 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:05.616 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:05.616 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:05.617 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:05.617 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:05.617 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:05.617 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:05.618 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:05.619 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:05.619 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:05.619 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:05.619 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:05.620 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:05.620 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:05.620 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:05.620 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.621 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:05.621 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:05.621 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:05.621 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:05.621 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.773406_193.1", 2021-09-06T13:24:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:05.622 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.773406Z", 2021-09-06T13:24:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.623 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:05.624 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:05.624 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:05.624 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:05.624 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:05.625 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:05.625 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:05.626 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.773406_193.1 2021-09-06T13:24:10.627 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:10.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:10.628 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:10.717 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:10.718 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:10.825 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:10.825 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:10.826 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:10.826 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:10.826 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:10.827 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:10.827 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:10.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:10.827 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:10.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:10.828 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:10.828 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:10.828 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:10.828 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:10.829 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:10.829 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:10.829 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:10.829 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:10.830 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:10.830 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.830 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:10.830 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:10.831 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:10.831 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:10.831 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:10.831 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:10.832 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:10.832 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:10.832 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:10.832 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:10.832 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:10.833 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:10.833 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:10.833 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:10.834 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:10.835 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:10.835 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:10.835 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:10.835 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.836 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:10.836 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:10.836 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:10.836 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:10.836 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:10.837 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.655178_183.1", 2021-09-06T13:24:10.837 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:10.837 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:10.837 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:10.838 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.655178Z", 2021-09-06T13:24:10.838 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:10.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.838 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.838 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:10.839 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:10.839 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:10.839 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:10.839 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:10.840 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:10.840 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:10.841 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.655178_183.1 2021-09-06T13:24:15.842 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:15.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:15.844 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:15.909 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:15.909 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:16.531 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:16.532 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:16.532 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:16.532 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:16.533 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:16.533 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:16.533 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:16.533 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:16.533 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:16.534 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:16.534 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:16.534 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:16.534 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:16.535 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:16.535 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:16.535 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:16.535 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:16.535 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:16.536 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:16.536 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.536 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:16.536 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:16.537 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:16.537 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:16.537 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:16.537 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:16.537 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:16.538 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:16.538 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:16.538 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:16.538 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:16.539 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.539 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:16.539 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:16.539 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:16.539 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:16.540 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:16.540 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:16.540 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:16.540 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:16.541 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:16.541 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:16.541 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:16.541 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.541 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:16.542 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:16.542 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:16.542 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:16.542 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:16.542 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.650194_182.1", 2021-09-06T13:24:16.543 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:16.543 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:16.543 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:16.543 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.650194Z", 2021-09-06T13:24:16.544 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:16.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.544 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.544 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:16.544 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:16.545 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:16.545 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:16.545 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:16.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:16.545 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:16.546 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.650194_182.1 2021-09-06T13:24:21.547 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:21.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:21.548 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:21.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:21.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:21.674 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:21.674 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:21.675 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:21.676 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:21.677 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:21.678 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:21.679 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:21.680 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:21.681 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:21.682 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:21.683 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:21.684 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.457681_169.1", 2021-09-06T13:24:21.685 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.457681Z", 2021-09-06T13:24:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:21.686 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:21.687 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:21.688 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:21.688 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:21.689 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.457681_169.1 2021-09-06T13:24:26.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:26.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:26.693 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:26.744 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:26.744 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:26.819 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:26.819 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:26.820 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:26.820 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:26.820 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:26.820 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:26.821 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:26.821 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:26.821 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:26.821 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:26.822 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:26.822 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:26.822 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:26.822 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:26.822 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:26.823 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:26.823 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:26.823 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:26.823 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:26.824 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.824 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:26.824 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:26.824 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:26.824 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:26.825 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:26.825 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:26.825 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:26.825 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:26.826 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:26.826 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:26.826 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:26.826 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.827 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:26.827 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:26.827 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:26.827 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:26.827 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:26.828 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:26.828 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:26.828 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:26.828 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:26.829 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:26.829 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:26.829 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.829 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:26.829 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:26.830 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:26.830 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:26.830 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:26.830 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:24:26.831 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:26.831 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:26.831 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:26.831 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:24:26.832 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:26.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.832 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.832 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:26.833 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:26.833 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:26.833 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:26.833 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:26.833 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:26.834 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:26.835 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:24:31.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:31.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:31.839 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:31.932 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:31.933 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:32.302 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:32.303 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:32.304 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:32.305 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:32.306 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:32.307 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:32.308 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:32.309 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:32.310 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:32.311 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:32.312 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:32.313 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.369173_219.1", 2021-09-06T13:24:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.369173Z", 2021-09-06T13:24:32.314 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:32.315 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:32.316 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:32.316 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:32.318 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932662.369173_219.1 2021-09-06T13:24:37.319 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:37.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:37.321 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:37.394 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:37.395 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:37.502 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:37.502 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:37.502 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:37.503 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:37.503 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:37.503 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:37.503 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:37.503 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:37.504 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:37.504 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:37.504 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:37.505 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:37.505 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:37.505 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:37.506 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:37.506 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:37.506 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:37.506 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:37.506 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:37.507 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.507 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:37.507 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:37.507 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:37.508 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:37.508 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:37.508 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:37.508 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:37.509 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:37.509 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:37.509 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:37.510 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:37.510 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.510 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:37.510 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:37.510 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:37.511 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:37.511 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:37.511 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:37.511 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:37.512 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:37.512 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:37.512 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:37.512 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:37.512 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.513 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:37.513 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:37.513 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:37.513 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:37.514 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:37.514 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:24:37.514 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:37.514 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:37.514 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:37.515 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:24:37.515 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:37.515 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.516 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.516 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:37.516 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:37.516 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:37.517 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:37.517 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:37.517 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:37.517 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:37.519 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:24:42.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:42.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:42.522 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:42.618 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:42.619 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:43.224 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:43.225 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:43.225 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:43.225 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:43.225 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:43.225 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:43.226 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:43.226 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:43.226 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:43.226 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:43.227 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:43.227 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:43.227 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:43.227 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:43.228 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:43.228 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:43.228 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:43.228 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:43.228 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:43.229 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.229 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:43.229 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:43.229 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:43.230 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:43.230 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:43.230 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:43.230 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:43.231 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:43.231 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:43.231 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:43.231 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:43.231 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.232 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:43.232 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:43.232 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:43.232 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:43.233 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:43.233 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:43.233 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:43.233 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:43.234 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:43.234 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:43.234 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:43.234 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.234 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:43.235 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:43.235 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:43.235 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:43.235 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:43.236 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.128782_213.1", 2021-09-06T13:24:43.236 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:43.236 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:43.236 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:43.236 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.128782Z", 2021-09-06T13:24:43.237 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:43.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.237 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.237 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:43.238 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:43.238 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:43.238 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:43.238 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:43.238 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:43.239 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:43.240 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932662.128782_213.1 2021-09-06T13:24:48.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:48.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:48.243 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:48.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:48.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:48.973 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:48.974 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:48.974 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:48.974 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:48.975 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:48.975 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:48.975 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:48.975 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:48.976 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:48.977 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:48.977 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:48.977 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:48.977 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:48.978 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:48.978 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:48.978 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.978 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:48.979 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:48.979 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:48.979 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:48.979 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:48.979 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:48.980 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:48.980 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:48.980 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:48.980 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:48.981 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:48.981 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.981 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:48.982 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:48.982 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:48.982 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:48.982 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:48.983 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:48.983 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:48.983 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:48.983 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:48.984 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:48.984 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:48.984 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.984 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:48.984 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:48.985 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:48.985 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:48.985 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:48.985 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.476870_172.1", 2021-09-06T13:24:48.986 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:48.986 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:48.986 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:48.986 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.476870Z", 2021-09-06T13:24:48.986 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:48.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.987 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.987 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:48.987 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:48.988 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:48.988 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:48.988 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:48.988 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:48.988 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:48.989 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.476870_172.1 2021-09-06T13:24:53.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:53.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:53.991 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:54.039 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:54.040 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:54.115 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:54.168 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:54.169 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:54.170 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:54.171 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:54.172 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:54.173 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:54.174 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:54.176 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:54.177 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:54.178 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:54.179 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.483094_173.1", 2021-09-06T13:24:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.483094Z", 2021-09-06T13:24:54.180 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:54.181 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:54.182 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:54.182 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:54.184 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.483094_173.1 2021-09-06T13:24:59.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:59.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:24:59.187 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:24:59.234 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:24:59.235 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:24:59.306 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:24:59.307 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:24:59.307 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:24:59.307 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:24:59.307 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:24:59.308 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:24:59.308 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:59.308 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:59.308 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:24:59.309 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:59.309 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:24:59.309 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:59.309 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:59.309 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:24:59.310 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:59.310 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:24:59.310 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:59.310 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:24:59.311 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:59.311 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.311 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:59.311 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:59.311 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:24:59.312 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:59.312 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:59.312 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:24:59.312 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:59.313 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:59.313 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:59.313 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:24:59.313 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:59.314 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.314 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:59.314 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:59.314 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:24:59.315 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:59.315 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:59.315 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:24:59.315 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:59.316 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:59.316 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:59.316 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:24:59.316 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:59.317 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.317 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:59.317 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:24:59.317 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:24:59.318 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:24:59.318 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:24:59.318 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.595266_180.1", 2021-09-06T13:24:59.318 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:24:59.319 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:24:59.319 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:24:59.319 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.595266Z", 2021-09-06T13:24:59.319 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:24:59.319 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.320 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.320 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:24:59.320 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:24:59.320 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:24:59.320 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:24:59.321 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:24:59.321 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:24:59.321 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:24:59.323 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.595266_180.1 2021-09-06T13:25:04.324 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:04.325 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:04.326 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:25:04.411 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:04.412 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:04.640 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:25:04.640 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:25:04.640 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:25:04.641 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:25:04.641 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:25:04.641 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:04.641 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:04.642 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:04.642 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:25:04.642 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:04.642 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:25:04.642 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:04.643 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:04.643 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:25:04.643 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:04.643 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:25:04.644 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:04.644 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:25:04.644 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:04.644 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.644 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:04.645 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:04.645 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:25:04.645 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:04.645 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:04.645 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:25:04.646 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:04.646 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:04.646 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:04.646 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:25:04.647 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:04.647 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.647 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:04.647 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:04.647 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:25:04.648 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:04.648 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:04.648 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:25:04.648 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:04.649 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:04.649 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:04.649 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:25:04.649 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:04.649 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.650 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:04.650 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:04.650 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:25:04.650 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:04.650 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:04.651 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.516029_175.1", 2021-09-06T13:25:04.651 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:04.651 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:04.651 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:04.652 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.516029Z", 2021-09-06T13:25:04.652 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:04.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.652 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.652 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:25:04.653 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:04.653 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:25:04.653 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:25:04.653 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:25:04.653 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:04.654 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:25:04.655 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.516029_175.1 2021-09-06T13:25:09.656 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:09.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:09.658 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:25:09.703 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:09.703 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:09.773 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:25:09.773 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:25:09.773 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:25:09.774 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:25:09.774 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:25:09.774 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:09.774 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:09.775 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:09.775 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:25:09.775 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:09.775 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:25:09.775 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:09.776 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:09.776 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:25:09.776 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:09.776 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:25:09.776 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:09.777 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:25:09.777 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:09.777 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.777 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:09.777 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:09.778 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:25:09.778 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:09.778 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:09.778 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:25:09.779 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:09.779 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:09.779 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:09.779 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:25:09.779 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:09.780 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.780 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:09.780 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:09.780 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:09.781 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:09.782 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:25:09.782 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:09.782 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.782 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:25:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:09.783 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.721579_187.1", 2021-09-06T13:25:09.784 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:09.784 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:09.784 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:09.784 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.721579Z", 2021-09-06T13:25:09.784 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:09.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.785 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.785 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:25:09.785 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:09.785 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:25:09.786 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:25:09.786 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:25:09.786 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:09.786 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:25:09.788 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.721579_187.1 2021-09-06T13:25:14.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:14.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:14.791 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:25:14.869 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:14.870 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:15.399 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:25:15.400 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:25:15.400 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:25:15.400 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:25:15.400 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:25:15.401 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:15.401 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:15.401 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:15.401 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:25:15.402 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:15.402 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:25:15.402 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:15.402 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:15.403 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:25:15.403 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:15.403 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:25:15.403 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:15.403 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:25:15.404 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:15.404 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.404 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:15.404 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:15.404 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:25:15.405 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:15.405 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:15.405 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:25:15.405 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:15.405 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:15.406 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:15.406 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:25:15.406 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:15.406 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.407 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:15.407 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:15.407 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:25:15.407 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:15.407 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:15.408 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:25:15.408 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:15.408 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:15.408 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:15.408 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:25:15.409 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:15.409 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.409 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:15.409 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:15.410 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:25:15.410 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:15.410 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:15.410 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932661.713479_184.1", 2021-09-06T13:25:15.410 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:15.411 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:15.411 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:15.411 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:01.713479Z", 2021-09-06T13:25:15.411 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:15.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.412 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.412 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:25:15.412 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:15.412 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:25:15.413 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:25:15.413 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:25:15.413 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:15.413 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:25:15.414 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932661.713479_184.1 2021-09-06T13:25:20.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:20.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:25:20.417 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:25:20.464 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:20.464 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:20.538 INFO:teuthology.orchestra.run.smithi052.stdout:{ 2021-09-06T13:25:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "sync_status": { 2021-09-06T13:25:20.538 INFO:teuthology.orchestra.run.smithi052.stdout: "info": { 2021-09-06T13:25:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "status": "sync", 2021-09-06T13:25:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "num_shards": 4, 2021-09-06T13:25:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "period": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:20.539 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "markers": [ 2021-09-06T13:25:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:20.540 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 0, 2021-09-06T13:25:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932315.333383_35.1", 2021-09-06T13:25:20.541 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 1, 2021-09-06T13:25:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:45:15.333383Z", 2021-09-06T13:25:20.542 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 1, 2021-09-06T13:25:20.543 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "", 2021-09-06T13:25:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:20.544 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "0.000000", 2021-09-06T13:25:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.545 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 2, 2021-09-06T13:25:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:20.546 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932459.927055_112.1", 2021-09-06T13:25:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:20.547 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:47:39.927055Z", 2021-09-06T13:25:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.548 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: { 2021-09-06T13:25:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "key": 3, 2021-09-06T13:25:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "val": { 2021-09-06T13:25:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "state": 1, 2021-09-06T13:25:20.549 INFO:teuthology.orchestra.run.smithi052.stdout: "marker": "1_1630932662.268828_215.1", 2021-09-06T13:25:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "next_step_marker": "", 2021-09-06T13:25:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "total_entries": 0, 2021-09-06T13:25:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "pos": 0, 2021-09-06T13:25:20.550 INFO:teuthology.orchestra.run.smithi052.stdout: "timestamp": "2021-09-06T12:51:02.268828Z", 2021-09-06T13:25:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: "realm_epoch": 2 2021-09-06T13:25:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.551 INFO:teuthology.orchestra.run.smithi052.stdout: ] 2021-09-06T13:25:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: }, 2021-09-06T13:25:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "full_sync": { 2021-09-06T13:25:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "total": 1, 2021-09-06T13:25:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: "complete": 1 2021-09-06T13:25:20.552 INFO:teuthology.orchestra.run.smithi052.stdout: } 2021-09-06T13:25:20.553 INFO:teuthology.orchestra.run.smithi052.stdout:} 2021-09-06T13:25:20.554 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=3 master=1_1630934020.575123_295.1 target=1_1630932662.268828_215.1 2021-09-06T13:25:25.556 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... FAIL 2021-09-06T13:25:25.558 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.559 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:25.606 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:25.607 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:25.643 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:25:25.643 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "cd4a3c48-2757-43da-937d-8a5c7ed9b2f9", 2021-09-06T13:25:25.643 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "remove_zg", 2021-09-06T13:25:25.644 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "remove_zg", 2021-09-06T13:25:25.644 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "false", 2021-09-06T13:25:25.644 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [], 2021-09-06T13:25:25.644 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T13:25:25.645 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T13:25:25.645 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "", 2021-09-06T13:25:25.645 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [], 2021-09-06T13:25:25.645 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [], 2021-09-06T13:25:25.646 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "", 2021-09-06T13:25:25.646 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:25.646 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T13:25:25.646 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T13:25:25.647 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.647 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:25:25.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.649 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:25.692 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:25.692 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:25.726 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:25:25.727 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "db38fe97-e559-4d95-855d-173e52f6b726", 2021-09-06T13:25:25.727 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "remove", 2021-09-06T13:25:25.727 INFO:teuthology.orchestra.run.smithi041.stdout: "domain_root": "remove.rgw.meta:root", 2021-09-06T13:25:25.727 INFO:teuthology.orchestra.run.smithi041.stdout: "control_pool": "remove.rgw.control", 2021-09-06T13:25:25.728 INFO:teuthology.orchestra.run.smithi041.stdout: "gc_pool": "remove.rgw.log:gc", 2021-09-06T13:25:25.728 INFO:teuthology.orchestra.run.smithi041.stdout: "lc_pool": "remove.rgw.log:lc", 2021-09-06T13:25:25.728 INFO:teuthology.orchestra.run.smithi041.stdout: "log_pool": "remove.rgw.log", 2021-09-06T13:25:25.728 INFO:teuthology.orchestra.run.smithi041.stdout: "intent_log_pool": "remove.rgw.log:intent", 2021-09-06T13:25:25.728 INFO:teuthology.orchestra.run.smithi041.stdout: "usage_log_pool": "remove.rgw.log:usage", 2021-09-06T13:25:25.729 INFO:teuthology.orchestra.run.smithi041.stdout: "roles_pool": "remove.rgw.meta:roles", 2021-09-06T13:25:25.729 INFO:teuthology.orchestra.run.smithi041.stdout: "reshard_pool": "remove.rgw.log:reshard", 2021-09-06T13:25:25.729 INFO:teuthology.orchestra.run.smithi041.stdout: "user_keys_pool": "remove.rgw.meta:users.keys", 2021-09-06T13:25:25.729 INFO:teuthology.orchestra.run.smithi041.stdout: "user_email_pool": "remove.rgw.meta:users.email", 2021-09-06T13:25:25.730 INFO:teuthology.orchestra.run.smithi041.stdout: "user_swift_pool": "remove.rgw.meta:users.swift", 2021-09-06T13:25:25.730 INFO:teuthology.orchestra.run.smithi041.stdout: "user_uid_pool": "remove.rgw.meta:users.uid", 2021-09-06T13:25:25.730 INFO:teuthology.orchestra.run.smithi041.stdout: "otp_pool": "remove.rgw.otp", 2021-09-06T13:25:25.730 INFO:teuthology.orchestra.run.smithi041.stdout: "system_key": { 2021-09-06T13:25:25.731 INFO:teuthology.orchestra.run.smithi041.stdout: "access_key": "", 2021-09-06T13:25:25.731 INFO:teuthology.orchestra.run.smithi041.stdout: "secret_key": "" 2021-09-06T13:25:25.731 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.731 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_pools": [ 2021-09-06T13:25:25.732 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.732 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "default-placement", 2021-09-06T13:25:25.732 INFO:teuthology.orchestra.run.smithi041.stdout: "val": { 2021-09-06T13:25:25.732 INFO:teuthology.orchestra.run.smithi041.stdout: "index_pool": "remove.rgw.buckets.index", 2021-09-06T13:25:25.732 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": { 2021-09-06T13:25:25.733 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD": { 2021-09-06T13:25:25.733 INFO:teuthology.orchestra.run.smithi041.stdout: "data_pool": "remove.rgw.buckets.data" 2021-09-06T13:25:25.733 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.733 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.734 INFO:teuthology.orchestra.run.smithi041.stdout: "data_extra_pool": "remove.rgw.buckets.non-ec", 2021-09-06T13:25:25.734 INFO:teuthology.orchestra.run.smithi041.stdout: "index_type": 0 2021-09-06T13:25:25.734 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.734 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.735 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.735 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:25.735 INFO:teuthology.orchestra.run.smithi041.stdout: "notif_pool": "remove.rgw.log:notif" 2021-09-06T13:25:25.735 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:25:25.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.738 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:25.821 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:25.821 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:25.910 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:25:25.910 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:25.911 INFO:teuthology.orchestra.run.smithi041.stdout: "epoch": 6, 2021-09-06T13:25:25.911 INFO:teuthology.orchestra.run.smithi041.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T13:25:25.911 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": [], 2021-09-06T13:25:25.911 INFO:teuthology.orchestra.run.smithi041.stdout: "period_map": { 2021-09-06T13:25:25.912 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:25.912 INFO:teuthology.orchestra.run.smithi041.stdout: "zonegroups": [ 2021-09-06T13:25:25.912 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.912 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "cd4a3c48-2757-43da-937d-8a5c7ed9b2f9", 2021-09-06T13:25:25.913 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "remove_zg", 2021-09-06T13:25:25.913 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "remove_zg", 2021-09-06T13:25:25.913 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "false", 2021-09-06T13:25:25.913 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [], 2021-09-06T13:25:25.913 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T13:25:25.914 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T13:25:25.914 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "db38fe97-e559-4d95-855d-173e52f6b726", 2021-09-06T13:25:25.914 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [ 2021-09-06T13:25:25.914 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.915 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "db38fe97-e559-4d95-855d-173e52f6b726", 2021-09-06T13:25:25.915 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "remove", 2021-09-06T13:25:25.915 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [], 2021-09-06T13:25:25.915 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:25.916 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "false", 2021-09-06T13:25:25.916 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:25.916 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:25.916 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:25.917 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T13:25:25.917 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:25.917 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:25.917 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.918 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.918 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [ 2021-09-06T13:25:25.918 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.918 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default-placement", 2021-09-06T13:25:25.918 INFO:teuthology.orchestra.run.smithi041.stdout: "tags": [], 2021-09-06T13:25:25.919 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": [ 2021-09-06T13:25:25.919 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD" 2021-09-06T13:25:25.919 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:25:25.919 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "default-placement", 2021-09-06T13:25:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:25.920 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T13:25:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T13:25:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.921 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "test-zonegroup", 2021-09-06T13:25:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zonegroup", 2021-09-06T13:25:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "", 2021-09-06T13:25:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "true", 2021-09-06T13:25:25.922 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:25:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T13:25:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T13:25:25.923 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [ 2021-09-06T13:25:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:25:25.924 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone3", 2021-09-06T13:25:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8001" 2021-09-06T13:25:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.925 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:25.926 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T13:25:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:25.927 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:25:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone4", 2021-09-06T13:25:25.928 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2021-09-06T13:25:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:25.929 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "pubsub", 2021-09-06T13:25:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "false", 2021-09-06T13:25:25.930 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [ 2021-09-06T13:25:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "test-zone1" 2021-09-06T13:25:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:25.931 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T13:25:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:25.932 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:25:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:25.933 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:25.934 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:25.935 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:25.935 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T13:25:25.936 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:25.936 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:25.936 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.936 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.937 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:25:25.937 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone2", 2021-09-06T13:25:25.937 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:25.937 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T13:25:25.937 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.938 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:25.938 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:25.938 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:25.938 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:25.939 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:25.939 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "false", 2021-09-06T13:25:25.939 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:25.939 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:25.939 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.940 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.940 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [ 2021-09-06T13:25:25.940 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.940 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default-placement", 2021-09-06T13:25:25.941 INFO:teuthology.orchestra.run.smithi041.stdout: "tags": [], 2021-09-06T13:25:25.941 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": [ 2021-09-06T13:25:25.941 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD" 2021-09-06T13:25:25.941 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:25:25.941 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.942 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.942 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "default-placement", 2021-09-06T13:25:25.942 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:25.942 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T13:25:25.943 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T13:25:25.943 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.943 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.943 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:25.944 INFO:teuthology.orchestra.run.smithi041.stdout: "short_zone_ids": [ 2021-09-06T13:25:25.944 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.944 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:25:25.944 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3257390013 2021-09-06T13:25:25.944 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.945 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.945 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:25:25.945 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 2300744410 2021-09-06T13:25:25.945 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.946 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.946 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:25.946 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 554922684 2021-09-06T13:25:25.946 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.947 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.947 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:25:25.947 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3217630970 2021-09-06T13:25:25.947 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.948 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:25.948 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "db38fe97-e559-4d95-855d-173e52f6b726", 2021-09-06T13:25:25.948 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 4268238655 2021-09-06T13:25:25.948 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.948 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:25:25.949 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.949 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T13:25:25.949 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:25.949 INFO:teuthology.orchestra.run.smithi041.stdout: "period_config": { 2021-09-06T13:25:25.950 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T13:25:25.950 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T13:25:25.950 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T13:25:25.950 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T13:25:25.951 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T13:25:25.951 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T13:25:25.951 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.951 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T13:25:25.952 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T13:25:25.952 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T13:25:25.952 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T13:25:25.952 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T13:25:25.952 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T13:25:25.953 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:25.953 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:25.953 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:25.953 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_name": "test-realm", 2021-09-06T13:25:25.954 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:25:25.954 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:25:25.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:25.956 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:26.017 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:26.018 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:26.055 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:25:26.054+0000 7f81ab659380 0 zone id db38fe97-e559-4d95-855d-173e52f6b726 is not a part of zonegroup test-zonegroup 2021-09-06T13:25:26.055 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:25:26.054+0000 7f81ab659380 0 zone id db38fe97-e559-4d95-855d-173e52f6b726 is not a part of zonegroup default 2021-09-06T13:25:26.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.067 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:26.156 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:26.156 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:26.206 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:25:26.205+0000 7f7316b01380 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-09-06T13:25:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-09-06T13:25:26.212 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.213 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.214 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:26.282 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:26.283 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:26.383 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-09-06T13:25:26.384 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-09-06T13:25:26.450 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:25:26.451 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:25:26.680 INFO:teuthology.orchestra.run.smithi041.stdout:{ 2021-09-06T13:25:26.680 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:26.680 INFO:teuthology.orchestra.run.smithi041.stdout: "epoch": 7, 2021-09-06T13:25:26.680 INFO:teuthology.orchestra.run.smithi041.stdout: "predecessor_uuid": "b82eb02b-9c46-4fe5-be3a-c10996df6ad0", 2021-09-06T13:25:26.681 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_status": [], 2021-09-06T13:25:26.681 INFO:teuthology.orchestra.run.smithi041.stdout: "period_map": { 2021-09-06T13:25:26.681 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "588058fc-8215-429d-a24d-21295d48dfdc", 2021-09-06T13:25:26.681 INFO:teuthology.orchestra.run.smithi041.stdout: "zonegroups": [ 2021-09-06T13:25:26.682 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.682 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "test-zonegroup", 2021-09-06T13:25:26.682 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zonegroup", 2021-09-06T13:25:26.682 INFO:teuthology.orchestra.run.smithi041.stdout: "api_name": "", 2021-09-06T13:25:26.683 INFO:teuthology.orchestra.run.smithi041.stdout: "is_master": "true", 2021-09-06T13:25:26.683 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:26.683 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:25:26.683 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.683 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames": [], 2021-09-06T13:25:26.684 INFO:teuthology.orchestra.run.smithi041.stdout: "hostnames_s3website": [], 2021-09-06T13:25:26.684 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:26.684 INFO:teuthology.orchestra.run.smithi041.stdout: "zones": [ 2021-09-06T13:25:26.685 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.685 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:25:26.685 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone3", 2021-09-06T13:25:26.686 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:26.686 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8001" 2021-09-06T13:25:26.686 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.687 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:26.687 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:26.688 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:26.688 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:26.688 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:26.689 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T13:25:26.689 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:26.689 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:26.690 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.690 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.690 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:25:26.691 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone4", 2021-09-06T13:25:26.691 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:26.691 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi052.front.sepia.ceph.com:8001" 2021-09-06T13:25:26.692 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.692 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:26.692 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:26.693 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:26.693 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:26.694 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "pubsub", 2021-09-06T13:25:26.694 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "false", 2021-09-06T13:25:26.694 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [ 2021-09-06T13:25:26.695 INFO:teuthology.orchestra.run.smithi041.stdout: "test-zone1" 2021-09-06T13:25:26.695 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.695 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:26.696 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.696 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.697 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:26.697 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone1", 2021-09-06T13:25:26.697 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:26.697 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi041.front.sepia.ceph.com:8000" 2021-09-06T13:25:26.697 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.698 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:26.698 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:26.698 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:26.698 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:26.699 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:26.699 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "true", 2021-09-06T13:25:26.699 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:26.699 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:26.700 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.700 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.700 INFO:teuthology.orchestra.run.smithi041.stdout: "id": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:25:26.700 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "test-zone2", 2021-09-06T13:25:26.701 INFO:teuthology.orchestra.run.smithi041.stdout: "endpoints": [ 2021-09-06T13:25:26.701 INFO:teuthology.orchestra.run.smithi041.stdout: "http://smithi052.front.sepia.ceph.com:8000" 2021-09-06T13:25:26.701 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.701 INFO:teuthology.orchestra.run.smithi041.stdout: "log_meta": "false", 2021-09-06T13:25:26.701 INFO:teuthology.orchestra.run.smithi041.stdout: "log_data": "true", 2021-09-06T13:25:26.702 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_index_max_shards": 11, 2021-09-06T13:25:26.702 INFO:teuthology.orchestra.run.smithi041.stdout: "read_only": "false", 2021-09-06T13:25:26.702 INFO:teuthology.orchestra.run.smithi041.stdout: "tier_type": "", 2021-09-06T13:25:26.702 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from_all": "false", 2021-09-06T13:25:26.703 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_from": [], 2021-09-06T13:25:26.703 INFO:teuthology.orchestra.run.smithi041.stdout: "redirect_zone": "" 2021-09-06T13:25:26.703 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.703 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.703 INFO:teuthology.orchestra.run.smithi041.stdout: "placement_targets": [ 2021-09-06T13:25:26.704 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.704 INFO:teuthology.orchestra.run.smithi041.stdout: "name": "default-placement", 2021-09-06T13:25:26.704 INFO:teuthology.orchestra.run.smithi041.stdout: "tags": [], 2021-09-06T13:25:26.704 INFO:teuthology.orchestra.run.smithi041.stdout: "storage_classes": [ 2021-09-06T13:25:26.705 INFO:teuthology.orchestra.run.smithi041.stdout: "STANDARD" 2021-09-06T13:25:26.705 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:25:26.705 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.705 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.706 INFO:teuthology.orchestra.run.smithi041.stdout: "default_placement": "default-placement", 2021-09-06T13:25:26.706 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:26.706 INFO:teuthology.orchestra.run.smithi041.stdout: "sync_policy": { 2021-09-06T13:25:26.706 INFO:teuthology.orchestra.run.smithi041.stdout: "groups": [] 2021-09-06T13:25:26.707 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.707 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.707 INFO:teuthology.orchestra.run.smithi041.stdout: ], 2021-09-06T13:25:26.707 INFO:teuthology.orchestra.run.smithi041.stdout: "short_zone_ids": [ 2021-09-06T13:25:26.707 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.718 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "4561051f-a4da-4b2a-84e5-f27920586526", 2021-09-06T13:25:26.718 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3257390013 2021-09-06T13:25:26.719 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.719 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.719 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "8c2685c1-df19-46c2-8a62-6cb4561f09e2", 2021-09-06T13:25:26.719 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 2300744410 2021-09-06T13:25:26.720 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.720 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.720 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:26.720 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 554922684 2021-09-06T13:25:26.720 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.721 INFO:teuthology.orchestra.run.smithi041.stdout: { 2021-09-06T13:25:26.721 INFO:teuthology.orchestra.run.smithi041.stdout: "key": "b8f5da72-5870-48fa-9c1e-4ed10a947347", 2021-09-06T13:25:26.721 INFO:teuthology.orchestra.run.smithi041.stdout: "val": 3217630970 2021-09-06T13:25:26.721 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.722 INFO:teuthology.orchestra.run.smithi041.stdout: ] 2021-09-06T13:25:26.722 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.722 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zonegroup": "test-zonegroup", 2021-09-06T13:25:26.723 INFO:teuthology.orchestra.run.smithi041.stdout: "master_zone": "a488f9d2-4d4c-49b4-b839-c5d73c4d316e", 2021-09-06T13:25:26.723 INFO:teuthology.orchestra.run.smithi041.stdout: "period_config": { 2021-09-06T13:25:26.724 INFO:teuthology.orchestra.run.smithi041.stdout: "bucket_quota": { 2021-09-06T13:25:26.724 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T13:25:26.725 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T13:25:26.725 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T13:25:26.725 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T13:25:26.725 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T13:25:26.726 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.726 INFO:teuthology.orchestra.run.smithi041.stdout: "user_quota": { 2021-09-06T13:25:26.726 INFO:teuthology.orchestra.run.smithi041.stdout: "enabled": false, 2021-09-06T13:25:26.726 INFO:teuthology.orchestra.run.smithi041.stdout: "check_on_raw": false, 2021-09-06T13:25:26.727 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size": -1, 2021-09-06T13:25:26.727 INFO:teuthology.orchestra.run.smithi041.stdout: "max_size_kb": 0, 2021-09-06T13:25:26.727 INFO:teuthology.orchestra.run.smithi041.stdout: "max_objects": -1 2021-09-06T13:25:26.727 INFO:teuthology.orchestra.run.smithi041.stdout: } 2021-09-06T13:25:26.727 INFO:teuthology.orchestra.run.smithi041.stdout: }, 2021-09-06T13:25:26.728 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_id": "9b968e6d-c378-4ac6-abc0-926118720ae1", 2021-09-06T13:25:26.728 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_name": "test-realm", 2021-09-06T13:25:26.728 INFO:teuthology.orchestra.run.smithi041.stdout: "realm_epoch": 2 2021-09-06T13:25:26.728 INFO:teuthology.orchestra.run.smithi041.stdout:} 2021-09-06T13:25:26.730 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-09-06T13:25:26.731 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-46 2021-09-06T13:25:27.397 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=hmrhlz-47 2021-09-06T13:25:27.485 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=hmrhlz-48 2021-09-06T13:25:27.559 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-49 2021-09-06T13:25:27.561 INFO:tasks.rgw.c1.client.0.smithi041.stdout:*** Caught signal (Segmentation fault) ** 2021-09-06T13:25:27.561 INFO:tasks.rgw.c1.client.0.smithi041.stdout: in thread 7fc8d3b64700 thread_name:radosgw 2021-09-06T13:25:27.561 INFO:tasks.rgw.c1.client.0.smithi041.stdout: ceph version 17.0.0-7472-g0eb1a794 (0eb1a7943dd70e2a0b3086ea680284137a187e73) quincy (dev) 2021-09-06T13:25:27.561 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9a475bb20] 2021-09-06T13:25:27.562 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fc9a735876e] 2021-09-06T13:25:27.562 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fc9a73764d4] 2021-09-06T13:25:27.562 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 4: /lib64/libradosgw.so.2(+0x533c3d) [0x7fc9a72a9c3d] 2021-09-06T13:25:27.562 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 5: /lib64/libradosgw.so.2(+0x535220) [0x7fc9a72ab220] 2021-09-06T13:25:27.563 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 6: /lib64/libradosgw.so.2(+0x53537c) [0x7fc9a72ab37c] 2021-09-06T13:25:27.563 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 7: make_fcontext() 2021-09-06T13:25:27.563 INFO:tasks.rgw.c1.client.0.smithi041.stdout:2021-09-06T13:25:27.559+0000 7fc8d3b64700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T13:25:27.563 INFO:tasks.rgw.c1.client.0.smithi041.stdout: in thread 7fc8d3b64700 thread_name:radosgw 2021-09-06T13:25:27.563 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.564 INFO:tasks.rgw.c1.client.0.smithi041.stdout: ceph version 17.0.0-7472-g0eb1a794 (0eb1a7943dd70e2a0b3086ea680284137a187e73) quincy (dev) 2021-09-06T13:25:27.564 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9a475bb20] 2021-09-06T13:25:27.564 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fc9a735876e] 2021-09-06T13:25:27.564 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fc9a73764d4] 2021-09-06T13:25:27.565 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 4: /lib64/libradosgw.so.2(+0x533c3d) [0x7fc9a72a9c3d] 2021-09-06T13:25:27.565 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 5: /lib64/libradosgw.so.2(+0x535220) [0x7fc9a72ab220] 2021-09-06T13:25:27.565 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 6: /lib64/libradosgw.so.2(+0x53537c) [0x7fc9a72ab37c] 2021-09-06T13:25:27.565 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 7: make_fcontext() 2021-09-06T13:25:27.566 INFO:tasks.rgw.c1.client.0.smithi041.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T13:25:27.566 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.578 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 0> 2021-09-06T13:25:27.559+0000 7fc8d3b64700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T13:25:27.579 INFO:tasks.rgw.c1.client.0.smithi041.stdout: in thread 7fc8d3b64700 thread_name:radosgw 2021-09-06T13:25:27.579 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.579 INFO:tasks.rgw.c1.client.0.smithi041.stdout: ceph version 17.0.0-7472-g0eb1a794 (0eb1a7943dd70e2a0b3086ea680284137a187e73) quincy (dev) 2021-09-06T13:25:27.579 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9a475bb20] 2021-09-06T13:25:27.580 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fc9a735876e] 2021-09-06T13:25:27.580 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fc9a73764d4] 2021-09-06T13:25:27.580 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 4: /lib64/libradosgw.so.2(+0x533c3d) [0x7fc9a72a9c3d] 2021-09-06T13:25:27.580 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 5: /lib64/libradosgw.so.2(+0x535220) [0x7fc9a72ab220] 2021-09-06T13:25:27.580 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 6: /lib64/libradosgw.so.2(+0x53537c) [0x7fc9a72ab37c] 2021-09-06T13:25:27.581 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 7: make_fcontext() 2021-09-06T13:25:27.581 INFO:tasks.rgw.c1.client.0.smithi041.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T13:25:27.581 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.596 INFO:tasks.rgw.c1.client.0.smithi041.stdout: -10> 2021-09-06T13:25:27.559+0000 7fc8d3b64700 -1 *** Caught signal (Segmentation fault) ** 2021-09-06T13:25:27.596 INFO:tasks.rgw.c1.client.0.smithi041.stdout: in thread 7fc8d3b64700 thread_name:radosgw 2021-09-06T13:25:27.597 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.597 INFO:tasks.rgw.c1.client.0.smithi041.stdout: ceph version 17.0.0-7472-g0eb1a794 (0eb1a7943dd70e2a0b3086ea680284137a187e73) quincy (dev) 2021-09-06T13:25:27.597 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 1: /lib64/libpthread.so.0(+0x12b20) [0x7fc9a475bb20] 2021-09-06T13:25:27.597 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2: (rgw_log_op(rgw::sal::Store*, RGWREST*, req_state*, std::__cxx11::basic_string, std::allocator > const&, OpsLogSocket*)+0x5be) [0x7fc9a735876e] 2021-09-06T13:25:27.598 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 3: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0x1864) [0x7fc9a73764d4] 2021-09-06T13:25:27.598 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 4: /lib64/libradosgw.so.2(+0x533c3d) [0x7fc9a72a9c3d] 2021-09-06T13:25:27.598 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 5: /lib64/libradosgw.so.2(+0x535220) [0x7fc9a72ab220] 2021-09-06T13:25:27.598 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 6: /lib64/libradosgw.so.2(+0x53537c) [0x7fc9a72ab37c] 2021-09-06T13:25:27.599 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 7: make_fcontext() 2021-09-06T13:25:27.599 INFO:tasks.rgw.c1.client.0.smithi041.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-09-06T13:25:27.599 INFO:tasks.rgw.c1.client.0.smithi041.stdout: 2021-09-06T13:25:27.755 INFO:tasks.rgw.c1.client.0.smithi041.stderr:daemon-helper: command crashed with signal 11 2021-09-06T13:25:31.327 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-09-06T13:25:36.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~6s 2021-09-06T13:25:42.539 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~11s 2021-09-06T13:25:48.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~17s 2021-09-06T13:25:53.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~22s 2021-09-06T13:25:59.356 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~28s 2021-09-06T13:26:04.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~34s 2021-09-06T13:26:10.567 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~39s 2021-09-06T13:26:16.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~45s 2021-09-06T13:26:21.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~50s 2021-09-06T13:26:27.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~56s 2021-09-06T13:26:32.990 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~62s 2021-09-06T13:26:38.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~67s 2021-09-06T13:26:44.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~73s 2021-09-06T13:26:49.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~78s 2021-09-06T13:26:55.416 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~84s 2021-09-06T13:26:58.253 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ERROR 2021-09-06T13:26:58.254 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-50 2021-09-06T13:27:01.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~90s 2021-09-06T13:27:06.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~95s 2021-09-06T13:27:12.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~101s 2021-09-06T13:27:17.838 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~107s 2021-09-06T13:27:23.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~112s 2021-09-06T13:27:29.049 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~118s 2021-09-06T13:27:34.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~123s 2021-09-06T13:27:40.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~129s 2021-09-06T13:27:45.867 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~135s 2021-09-06T13:27:51.473 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~140s 2021-09-06T13:27:57.079 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~146s 2021-09-06T13:28:02.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~151s 2021-09-06T13:28:08.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~157s 2021-09-06T13:28:13.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~163s 2021-09-06T13:28:19.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~168s 2021-09-06T13:28:25.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~174s 2021-09-06T13:28:27.162 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ERROR 2021-09-06T13:28:27.164 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-51 2021-09-06T13:28:30.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~179s 2021-09-06T13:28:36.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~185s 2021-09-06T13:28:41.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~191s 2021-09-06T13:28:47.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~196s 2021-09-06T13:28:53.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~202s 2021-09-06T13:28:58.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~207s 2021-09-06T13:29:04.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~213s 2021-09-06T13:29:09.690 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ERROR 2021-09-06T13:29:09.691 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-52 2021-09-06T13:29:09.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~219s 2021-09-06T13:29:15.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~224s 2021-09-06T13:29:21.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~230s 2021-09-06T13:29:26.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~235s 2021-09-06T13:29:32.388 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~241s 2021-09-06T13:29:37.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~247s 2021-09-06T13:29:43.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~252s 2021-09-06T13:29:49.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~258s 2021-09-06T13:29:54.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~263s 2021-09-06T13:30:00.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~269s 2021-09-06T13:30:06.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~275s 2021-09-06T13:30:11.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~280s 2021-09-06T13:30:17.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~286s 2021-09-06T13:30:22.839 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~292s 2021-09-06T13:30:28.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~297s 2021-09-06T13:30:33.126 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ERROR 2021-09-06T13:30:33.127 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-53 2021-09-06T13:30:34.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~303s 2021-09-06T13:30:34.052 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-09-06T13:30:34.656 INFO:tasks.ceph.c1.osd.0:Sent signal 15 2021-09-06T13:30:34.659 INFO:tasks.ceph.c1.osd.1:Sent signal 15 2021-09-06T13:30:34.659 INFO:tasks.ceph.c1.osd.2:Sent signal 15 2021-09-06T13:30:34.660 INFO:tasks.ceph.c1.mon.a:Sent signal 15 2021-09-06T13:30:34.660 INFO:tasks.rgw.c1.client.1:Sent signal 15 2021-09-06T13:30:34.661 INFO:tasks.ceph.c1.mgr.x:Sent signal 15 2021-09-06T13:30:34.662 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T13:30:34.657+0000 7fd7c1145700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 0 (PID: 31655) UID: 0 2021-09-06T13:30:34.662 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T13:30:34.657+0000 7fd7c1145700 -1 osd.0 62 *** Got signal Terminated *** 2021-09-06T13:30:34.662 INFO:tasks.ceph.c1.osd.0.smithi041.stderr:2021-09-06T13:30:34.657+0000 7fd7c1145700 -1 osd.0 62 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T13:30:34.663 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T13:30:34.659+0000 7f1ba01e9700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 1 (PID: 31656) UID: 0 2021-09-06T13:30:34.663 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T13:30:34.659+0000 7f1ba01e9700 -1 osd.1 62 *** Got signal Terminated *** 2021-09-06T13:30:34.664 INFO:tasks.ceph.c1.osd.1.smithi041.stderr:2021-09-06T13:30:34.659+0000 7f1ba01e9700 -1 osd.1 62 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T13:30:34.664 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T13:30:34.659+0000 7fbbc6f39700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster c1 -i 2 (PID: 31660) UID: 0 2021-09-06T13:30:34.665 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T13:30:34.659+0000 7fbbc6f39700 -1 osd.2 62 *** Got signal Terminated *** 2021-09-06T13:30:34.665 INFO:tasks.ceph.c1.osd.2.smithi041.stderr:2021-09-06T13:30:34.659+0000 7fbbc6f39700 -1 osd.2 62 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-09-06T13:30:34.665 INFO:tasks.ceph.c1.mon.a.smithi041.stderr:2021-09-06T13:30:34.660+0000 7f4fddd8e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster c1 -i a (PID: 31335) UID: 0 2021-09-06T13:30:34.666 INFO:tasks.ceph.c1.mon.a.smithi041.stderr:2021-09-06T13:30:34.660+0000 7f4fddd8e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-09-06T13:30:34.666 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T13:30:34.660+0000 7f89fd7a1700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 39802) UID: 0 2021-09-06T13:30:34.666 INFO:tasks.rgw.c1.client.1.smithi041.stdout:2021-09-06T13:30:34.660+0000 7f8a0e641440 -1 shutting down 2021-09-06T13:31:53.596 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ERROR 2021-09-06T13:31:53.597 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-54 2021-09-06T13:33:21.230 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ERROR 2021-09-06T13:33:21.231 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=hmrhlz-55 2021-09-06T13:34:07.575 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ERROR 2021-09-06T13:34:07.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:34:07.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T13:34:07.579 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T13:34:07.629 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T13:34:07.630 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T13:39:07.632 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:39:07.630+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:44:07.633 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:44:07.631+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:49:07.634 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:49:07.631+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:54:07.634 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:54:07.632+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T13:59:07.636 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T13:59:07.633+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:04:07.636 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:04:07.634+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:09:07.637 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:09:07.634+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:14:07.638 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:14:07.634+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:19:07.638 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:19:07.635+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:24:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T14:24:07.641 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:24:07.635+0000 7fd866dc8380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:24:07.642 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T14:24:07.643 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... FAIL 2021-09-06T14:24:07.644 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-09-06T14:24:07.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T14:24:07.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T14:24:07.648 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T14:24:07.699 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T14:24:07.700 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T14:29:07.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:29:07.699+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:34:07.703 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:34:07.700+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:39:07.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:39:07.700+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:44:07.704 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:44:07.701+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:49:07.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:49:07.700+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:54:07.705 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:54:07.702+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T14:59:07.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T14:59:07.702+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:04:07.706 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:04:07.703+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:09:07.707 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:09:07.704+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:14:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T15:14:07.711 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:14:07.704+0000 7f8f542ff380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:14:07.711 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T15:14:07.713 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... FAIL 2021-09-06T15:14:07.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T15:14:07.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T15:14:07.716 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T15:14:07.767 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T15:14:07.768 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T15:19:07.770 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:19:07.767+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:24:07.771 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:24:07.768+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:29:07.770 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:29:07.768+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:34:07.772 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:34:07.769+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:39:07.771 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:39:07.769+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:44:07.772 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:44:07.769+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:49:07.774 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:49:07.769+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:54:07.773 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:54:07.771+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T15:59:07.773 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T15:59:07.771+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:04:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T16:04:07.777 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:04:07.771+0000 7f2b91a62380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:04:07.777 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T16:04:07.779 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... FAIL 2021-09-06T16:04:07.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:04:07.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:04:07.781 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T16:04:07.833 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T16:04:07.833 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T16:09:07.836 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:09:07.833+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:14:07.836 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:14:07.834+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:19:07.836 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:19:07.833+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:24:07.837 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:24:07.835+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:29:07.838 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:29:07.836+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:34:07.838 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:34:07.836+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:39:07.839 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:39:07.837+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:44:07.840 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:44:07.838+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:49:07.840 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:49:07.838+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:54:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T16:54:07.844 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:54:07.838+0000 7f1549eed380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T16:54:07.844 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T16:54:07.846 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... FAIL 2021-09-06T16:54:07.847 INFO:tasks.rgw_multisite_tests:test s3 notification filter on master ... ok 2021-09-06T16:54:07.848 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_object_timing ... ok 2021-09-06T16:54:07.848 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-09-06T16:54:07.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:54:07.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T16:54:07.850 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T16:54:07.902 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T16:54:07.903 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T16:59:07.906 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T16:59:07.903+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:04:07.906 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:04:07.903+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:09:07.907 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:09:07.904+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:14:07.908 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:14:07.906+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:19:07.909 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:19:07.907+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:24:07.909 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:24:07.907+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:29:07.909 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:29:07.907+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:34:07.909 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:34:07.907+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:39:07.910 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:39:07.908+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:44:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T17:44:07.913 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:44:07.908+0000 7fefe8292380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:44:07.914 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T17:44:07.915 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... FAIL 2021-09-06T17:44:07.916 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:44:07.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T17:44:07.918 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T17:44:07.971 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T17:44:07.972 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T17:49:07.974 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:49:07.973+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:54:07.975 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:54:07.973+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T17:59:07.975 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T17:59:07.973+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:04:07.976 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:04:07.974+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:09:07.977 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:09:07.974+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:14:07.976 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:14:07.974+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:19:07.977 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:19:07.975+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:24:07.979 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:24:07.975+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:29:07.978 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:29:07.976+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:34:07.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T18:34:07.986 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:34:07.977+0000 7f87bba8a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:34:07.986 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T18:34:07.987 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... FAIL 2021-09-06T18:34:07.988 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T18:34:07.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T18:34:07.990 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T18:34:08.043 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T18:34:08.043 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T18:39:08.046 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:39:08.044+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:44:08.048 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:44:08.044+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:49:08.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:49:08.045+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:54:08.047 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:54:08.044+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T18:59:08.048 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T18:59:08.045+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:04:08.048 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:04:08.046+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:09:08.049 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:09:08.047+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:14:08.050 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:14:08.048+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:19:08.051 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:19:08.049+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:24:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T19:24:08.066 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:24:08.048+0000 7f170c57e380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:24:08.067 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T19:24:08.068 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... FAIL 2021-09-06T19:24:08.069 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T19:24:08.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T19:24:08.071 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T19:24:08.123 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T19:24:08.124 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T19:29:08.126 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:29:08.124+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:34:08.127 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:34:08.124+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:39:08.127 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:39:08.125+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:44:08.129 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:44:08.126+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:49:08.128 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:49:08.126+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:54:08.129 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:54:08.127+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T19:59:08.129 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T19:59:08.126+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:04:08.129 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:04:08.128+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:09:08.129 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:09:08.128+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:14:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T20:14:08.133 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:14:08.128+0000 7f6060ea6380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:14:08.133 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T20:14:08.135 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... FAIL 2021-09-06T20:14:08.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T20:14:08.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T20:14:08.137 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T20:14:08.189 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T20:14:08.189 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T20:19:08.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:19:08.190+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:24:08.191 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:24:08.191+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:29:08.193 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:29:08.191+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:34:08.192 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:34:08.191+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:39:08.195 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:39:08.192+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:44:08.194 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:44:08.193+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:49:08.195 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:49:08.193+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:54:08.195 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:54:08.194+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T20:59:08.196 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T20:59:08.195+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:04:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T21:04:08.205 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:04:08.195+0000 7f6e86afa380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:04:08.205 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T21:04:08.212 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... FAIL 2021-09-06T21:04:08.213 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:04:08.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:04:08.214 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T21:04:08.269 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T21:04:08.269 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T21:09:08.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:09:08.270+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:14:08.272 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:14:08.271+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:19:08.273 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:19:08.271+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:24:08.273 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:24:08.272+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:29:08.274 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:29:08.272+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:34:08.275 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:34:08.272+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:39:08.277 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:39:08.274+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:44:08.276 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:44:08.274+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:49:08.277 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:49:08.274+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:54:08.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T21:54:08.280 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:54:08.275+0000 7fba0deaf380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T21:54:08.281 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T21:54:08.282 INFO:tasks.rgw_multisite_tests:test radosgw-admin commands ... FAIL 2021-09-06T21:54:08.283 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:54:08.284 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T21:54:08.285 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T21:54:08.337 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T21:54:08.337 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T21:59:08.340 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T21:59:08.337+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:04:08.342 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:04:08.338+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:09:08.342 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:09:08.338+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:14:08.342 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:14:08.339+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:19:08.343 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:19:08.340+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:24:08.344 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:24:08.340+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:29:08.343 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:29:08.341+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:34:08.343 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:34:08.341+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:39:08.343 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:39:08.341+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:44:08.343 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:44:08.341+0000 7f702683a380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:44:08.370 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T22:44:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T22:44:08.374 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... FAIL 2021-09-06T22:44:08.375 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T22:44:08.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T22:44:08.376 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T22:44:08.428 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T22:44:08.428 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T22:49:08.430 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:49:08.428+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:54:08.431 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:54:08.430+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T22:59:08.431 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T22:59:08.430+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:04:08.432 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:04:08.430+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:09:08.433 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:09:08.431+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:14:08.433 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:14:08.432+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:19:08.434 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:19:08.432+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:24:08.434 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:24:08.433+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:29:08.435 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:29:08.434+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:34:08.435 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:34:08.434+0000 7f4bf6c2b380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:34:08.436 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-06T23:34:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-06T23:34:08.459 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... FAIL 2021-09-06T23:34:08.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T23:34:08.460 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-06T23:34:08.461 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-06T23:34:08.512 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-06T23:34:08.513 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-06T23:39:08.516 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:39:08.514+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:44:08.516 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:44:08.514+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:49:08.517 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:49:08.516+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:54:08.518 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:54:08.516+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-06T23:59:08.520 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-06T23:59:08.517+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:04:08.520 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-07T00:04:08.517+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:09:08.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-07T00:09:08.518+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:14:08.521 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-07T00:14:08.519+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:19:08.522 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-07T00:19:08.520+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:24:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-09-07T00:24:08.525 INFO:teuthology.orchestra.run.smithi041.stderr:2021-09-07T00:24:08.520+0000 7fbad31d1380 0 monclient(hunting): authenticate timed out after 300 2021-09-07T00:24:08.525 INFO:teuthology.orchestra.run.smithi041.stderr:failed to fetch mon config (--no-mon-config to skip) 2021-09-07T00:24:08.527 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... FAIL 2021-09-07T00:24:08.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-07T00:24:08.528 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-09-07T00:24:08.529 DEBUG:teuthology.orchestra.run.smithi041:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-09-07T00:24:08.581 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setuser ceph since I am not root 2021-09-07T00:24:08.581 INFO:teuthology.orchestra.run.smithi041.stderr:ignoring --setgroup ceph since I am not root 2021-09-07T00:25:31.204 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-09-07T00:25:31.205 DEBUG:teuthology.task.console_log:Killing console logger for smithi041 2021-09-07T00:25:31.206 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2021-09-07T00:25:31.207 DEBUG:teuthology.exit:Finished running handlers